Understanding os.fork and Queue.Queue

Question

I wanted to implement a simple python program using parallel execution. This is an I / O limitation, so I decided that threads would match (as opposed to processes). After reading the documentation for the queue and the plug, I thought something like the following might work.

q = Queue.Queue() if os.fork(): # child while True: print q.get() else: # parent [q.put(x) for x in range(10)] 

However, the call to get () never returns. I thought it would return when another thread made a put () call. Using the streaming module, everything looks as I expected:

 q = Queue.Queue() def consume(q): while True: print q.get() worker = threading.Thread (target=consume, args=(q,)) worker.start() [q.put(x) for x in range(10)] 

I just don't understand why the fork approach does not do the same. What am I missing?

Answer 1

The POSIX fork system call creates a new process, not a new thread, inside the same address space:

The fork () function should create a new process. The new process (child process) must be an exact copy of the calling process (parent process), except as described below: [...]

So, Queue duplicated in the first example, and is not split between the parent and the child.

You can use multiprocessing.Queue or just use threads, as in your second example :)

By the way, using lists for side effects is not good practice for several reasons. Instead, you should use a for loop:

 for x in range(10): q.put(x) 

Answer 2

To share data between unrelated processes, you can use named pipes. Via os.open () funcion .. http://docs.python.org/2/library/os.html#os.open . You can simply name the channel as named_pipe = 'my_pipe', and in different python programs use os.open (named_pipe,), where the mode is WRONLY and so on. After that, you will create a FIFO to write to the pipe. Remember to close the handset and eliminate exceptions.

Answer 3

Fork creates a new process. The child and parent processes do not use the same queue: why the elements supplied by the parent process cannot be received by the child.