Asynchronous logging with python and mongodb

Question

I need to log specific actions of my web application (based on Python - SQL alchemy with Postgres), and I don’t want to either write log data to my Postgres database (why fill it mainly with garbage?), Or use a log file (hard to find )

Ideally, I would like to drop everything into another database and do it asynchronously. With asynchronous logging, I don’t have to worry about the write operation failing and breaking code that does all the important things. Also, if I skip a few registration events, it probably doesn't matter much.

Mongo seems like a great solution as it is well suited for writing operations and is easy to configure.

The problem is that I was not able to find any python tools that cover my needs and in particular the asynchronous requirement.

Any thoughts?

Answer 1

Using a log collector daemon , such as Fluentd / Scribe / Flume , might be another solution.

This daemon runs on each application node and takes logs from application processes. They buffer logs and write data asynchronously to other systems, such as MongoDB / PostgreSQL / etc. Recording is performed in batches , so it is much more efficient than direct writing from applications.

Here are two links on how to use Fluentd from Python and how to put data in MongoDB.

• Fluentd: Import Data from Python Applications
• Fluentd: Store Apache Logs in MongoDB

Answer 2

Asynchronous logging in mongodb can be achieved by adding AsyncAppender to log4j, which will reference the real appender.

For a basic understanding, follow this http://wiki.python.org/jython/Log4jExample

log4mongo drags data into mongo.Python driver is available at http://log4mongo.org/display/PUB/Log4mongo+for+Python

adding AsyncAppender will make the asynchronous logging path

for reference only, equivalent log4j file

 <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"> <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/"> <appender name="MongoDB" class="org.log4mongo.MongoDbAppender" > <param name="hostname" value="127.0.0.1" /> <param name="port" value="27017" /> <param name="databaseName" value="LogDB"/> <param name="collectionName" value="Log" /> </appender> <appender name="ASYNC" class="org.apache.log4j.AsyncAppender"> <param name="BufferSize" value="50000"/> <appender-ref ref="MongoDB"/> </appender> <root> <level value="all"/> <appender-ref ref="ASYNC"/> </root> 

Answer 3

I am using log4mongo , available with pip:

 pip install log4mongo 

This allows you to use the default logging system. Example (extract from document):

 import logging from log4mongo.handlers import MongoHandler logger = logging.getLogger('test') logger.addHandler(MongoHandler(host='localhost')) logger.warning('test')