How to connect to a single file with simultaneous processes in Django without exclusive locks

Question

Considering the Django application, which runs on several servers simultaneously, how can this application be registered in one common log file (in a network share) without saving this file permanently in exclusive mode?

This situation applies to Django applications hosted on Windows Azure websites when you want to use the log stream.

In this sample project , I am trying to use ConcurrentLogHandler :

in settings.py :

'ConcurrentLogHandler':{ 'level': 'DEBUG', 'class': 'cloghandler.ConcurrentRotatingFileHandler', 'formatter': 'verbose', 'filename': os.getenv('LOGFILE', 'django.log') }, 

in views.py :

 from time import gmtime, strftime import logging from django.http import HttpResponse logger = logging.getLogger(__name__) def home(request): current_time = strftime("%Y-%m-%d %H:%M:%S", gmtime()) logger.info('home ' + current_time) return HttpResponse("Hello from Django! It is now " + current_time + ".\n") 

Logs are recorded, but the file does not seem to be cleared while the website is running. Also, if I try to read a file using FTP, I get this message: "550 The process cannot access the file because it is being used by another process."

If I stop the application, the file will be closed, and I can read the file and see all its logs.

I assume ConcurrentLogHandler will allow sharing of the log file. Is this assumption wrong? Do I need additional configuration? Is there an alternative?

Answer 1

An alternative would be for all Django logging to be sent to the queue (for example, to the Redis queue using something like this or multiprocessing.Queue ), and then one process reads the queue and writes the entries to the file. There are more moving parts, so this may or may not suit your needs, but it will eliminate the file conflict. See this post for more options when using logging from multiple processes.

You can also configure the socket server and use SocketHandler to send registration events from all Django processes to the server that writes to the file. Python docs contain a working example of such a server.

Answer 2

A scalable approach to pure logging (as when using only syslog components) can be implemented using syslog-ng

syslog-ng is syslog ++, so it will not break any syslog configurations and logging if you already have some. Installing on Ubuntu is simple:

 sudo apt-get install syslog-ng 

for installation on windows, see http://www.syslog.org/logged/running-syslog-ng-on-windows/

Several django applications or several servers are logged into syslog-ng, which uses UDP or TCP (depending on your configuration), to send it to the central syslog-ng server, which logs it to this computer. You can use basic regex to identify the application in syslog-ng and route it accordingly.

Django Log Configuration

 'syslog-verbose': { 'format': 'myapp %(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s' } 'ConcurrentLogHandler':{ 'level': 'DEBUG', 'class': 'logging.handlers.SysLogHandler', 'formatter': 'syslog-verbose', }, 

Note the line "myapp" added to the formatter to identify the syslog-ng application.

You can configure nginx, apache and other servers to log in to syslog-ng. For instance. for apache:

 CustomLog "| /usr/bin/logger -t 'apache' -u /var/run/apache_access_log.socket" combined 

Syslog-ng client configuration

Add this to the end of / etc / syslog -ng / syslog-ng.conf

 filter filter_loghostclient { program("^myapp$"); }; destination dest_loghostclient { tcp("destination_logserver.example.com" port (514)); }; log { source(s_all); filter(filter_loghostclient); destination(dest_loghostclient); }; source s_apache_access { #apache access logs getting written directly to a socket (as described above) unix-stream("/var/run/apache_access_log.socket" max-connections(512) keep-alive(yes)); }; log{ source(s_apache_access); destination(dest_loghostclient); }; filter f_apache_err { #Apache error logs program("apache") and level(err); }; log{ source(s_all); filter(f_apache_err); destination(dest_loghostclient); }; 

Syslog-ng aggregator configuration

Add the following to the syslog-ng configuration file: destination_logserver.example.com

 source src_loghostserver { tcp(port(514) keep-alive(yes) max_connections(1000)); }; destination dest_loghostserver { file("/var/log/myproject/request_\$R_YEAR\$R_MONTH\$R_DAY.log"); }; log { source(src_loghostserver); destination(dest_loghostserver); };