How to configure logging on dask distributed work resources?

Question

After updating a package distributed to version 1.15.0, my journal stops working.

I used logging.config.dictConfig to initialize the python logging tools, and previously these settings applied to all workers. But after the update, it no longer works.

If I do a dictConfig right before each log call for each worker, it works, but this is not the right solution.

So the question is, how does it initialize the logging for each employee before my calculation schedule starts and does it only once per employee?

UPDATE:

This hack worked on a dummy example, but did not change my system:

def init_logging(): # logging initializing happens here ... client = distributed.Client() client.map(lambda _: init_logging, client.ncores()) 

UPDATE 2:

After processing the documentation, this fixed the problem:

 client.run(init_logging) 

So the question is, is this the right way to solve this problem?

Answer 1

Starting from version 1.15.0, we now make fork workers from a clean process, so the changes you make before your process before calling Client() will not affect forked workers. For more information, find forkserver here: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Your decision to use Client.run looks good to me. Currently Client.run (since version 1.15.0) is the best way to call a function for all currently active workers.

Distributed systems

It is worth noting that here you configure clients that are bifurcated from one process on one computer. The trick you use above will not work in distributed setup. If people come to this question with a question about how to handle logging using Dask in a cluster context, I add this note.

Normally Dask does not move logs. Instead, typically, any mechanism you used to run Dask handles this. Job planners such as SGE / SLURM / Torque / PBS all do this. Cloud systems like YARN / Mesos / Marathon / Kubernetes all do this. The dask-ssh tool does this.