How can I create interprocess data structure in Python?

Question

I have a list (called requestRoster) containing dictionaries (called requests). Items in the query dictionary are things like "requestTime" and "thisURL". For instance:

[ {'thisURL': 'http://localhost/bikes', 'requestTime': datetime.datetime(2012, 10, 18, 0, 41, 34)}, {'thisURL': 'http://localhost/clothing', 'requestTime': datetime.datetime(2012, 10, 18, 0, 41, 35)} ] 

I use multiprocessing.Process to create a new process to issue each request.

I would like each process to update requestRoster by adding a response element to each request.

How can i do this?

I tried using multiprocessing.Manager () to create manager.list () and manager.Namespace (). I don't let me do what I want to do, I think because of this: http://docs.python.org/library/multiprocessing.html#multiprocessing.managers.SyncManager.list

I think I could use multiprocessing. Lock () -

• get mutex
• make a copy of requestRoster inside the process
• change localized query
• overwrite the globablised query list with localized
• let go of the mutex

... but it seems a bit complicated, and I wonder if I am missing something simpler. Asynchronous callback will be great.

Answer 1

It is better to avoid shared memory structures if possible. There is no reason for the processes to be recorded in the list of dictations themselves - instead, you could make the main process responsible for this and process only the URL selections for the processes.

I like concurrent.futures.<Process|Thread>PoolExecutor for this kind of thing.

Answer 2

I think this approach should work for you:

shipper:

 create logger_queue create logger process, initialize with logger_queue for each request create worker_pipe create worker process, initialize with send end of worker_pipe push receive end of worker_pipe over logger_queue 

Working:

 make request push response over connection 

Logger:

 while True for connection on logger_queue create new element in logging list link connection to new logging list element for each open connection poll for message if message store message to log close connection 

The registration process can also run any output routines that you need, so you don’t even have to worry about reading another process from a logged data set. Please note that the connection above refers to multiprocessing.Connection .

Answer 3

I managed to do this using multithreading instead of multiprocessing. Since workers are in the same process as the dispatcher, they can update requestRoster.