How to route log messages from a multiprocessor module to a file?

Question

In Python docs, I found that logging messages are possible in python scripts that use the multiprocessing module to create different processes.

import multiprocessing import logging logger = multiprocessing.log_to_stderr() logger.setLevel(logging.INFO) 

So logger.warning('doomed') will give me the message 'doomed' , but only to the console.

How can I switch log messages to a file?

Answer 1

Multiprocessing logging is not trivial, but it is not complicated. You will need the logging module.

The main thing is that the listener process receives all registration records through multiprocessor queues . This listener process then processes these entries using any internal handlers that you decide to use. In your case, you can use FileHandler or similar.

If you are in Python version 3.2 or later, you will have QueueHandler and QueueListener in the standard library. Quelistener is a listener process that I talked about before (yes, it has a start() method).

If you are in a previous version of Python, look at the link that basically shows how to rewrite them (and how to use them).

---

If you need links from an official document, see Writing to a single file from several processes from the Log section.

You may also be interested in this answer from the question How should I register when using multiprocessing in Python?