How to send large amounts of data from a forked process?

Question

I have a ctypes wrapper for a library. Unfortunately, this library is not 100% reliable (sometimes segfaults, etc.). Due to the way it is used, I want the shell to be sufficiently robust against a library crash.

The best way to do this seems to be to deploy the process and send the results from the child. I would like to do something in this direction:

r, w = os.pipe()
pid = os.fork()

if pid == 0:
    # child
    result = ctypes_fn()
    os.write(w, pickle.dumps(result))
    os.close(w)
else:
    # parent
    os.waitpid(pid, 0)
    result = os.read(r, 524288) # can be this big
    os.close(r)

    return pickle.loads(result)

This does not quite work. A broken process freezes during recording. Am I trying to send too much at once? Is there a simpler solution to this problem?

Answer 1

, , , , - . , , , , , , , , - . , .

os.waitpid , . , os.pipe - , ( , ).

Answer 2

, 64 . : :

• . zlib.compress .
• - (, mmap, memcache), .
• , . , . , .

Answer 3

, , :

#parent
while waitpid(pid, WNOHANG) == (0, 0):
    result = os.read(r, 1024)
    #sleep for a short time
#at this point the child process has ended 
#and you need the last bit of data from the pipe
result = os.read(r, 1024)
os.close(r)

Waitpid WNOHANG waitpid , . (0,0). , .