Multiprocessing

Question

I am trying to understand the following guide:

Better to inherit than pickle / unpickle

When using spawn or forkserver startup methods, many types of multiprocessing must be selected so that child processes can use them. However, you should generally avoid sending shared objects to other processes using pipes or queues. Instead, you should organize a program so that a process that needs access to a shared resource created elsewhere can inherit it from the ancestor process.

• What does it mean to "organize a program"?
• How can I share resources through inheritance?

I run windows, so new processes are spawned, does this mean that inherited processes can inherit?

Answer 1

1. What does it mean to "organize a program"?

This means that your program should work autonomously without any external resources. File sharing will give you problems with locking, memory sharing will either do the same, or it may lead to corruption due to several processes that modify data at the same time.

Here is an example of what would be a bad idea:

while some_queue_is_not_empty(): run_external_process(some_queue) def external_process(queue): item = queue.pop() # do processing here 

Versus:

 while some_queue_is_not_empty(): item = queue.pop() run_external_process(item) def external_process(item): # do processing here 

This way you can avoid blocking problems with the queue and / or corruption due to the fact that several processes receive the same element.

2. How can I share resources by inheriting?

On Windows you cannot. On Linux, you can use file descriptors opened by parents; on Windows, this will be a completely new process, so you have nothing from your parent except what was given.

Example copied from: http://rhodesmill.org/brandon/2010/python-multiprocessing-linux-windows/

 from multiprocessing import Process f = None def child(): print f if __name__ == '__main__': f = open('mp.py', 'r') p = Process(target=child) p.start() p.join() 

On Linux, you will get something like:

 $ python mp.py <open file 'mp.py', mode 'r' at 0xb7734ac8> 

On Windows, you will get:

 C:\Users\brandon\dev>python mp.py None