Profiling Python code using multiprocessing?

Question

I have a simple manufacturer sample created in part of my gui code. I am trying to profile only a specific consumer section to see if there is a chance for optimization. However, when I try to run the code using python -m cProfile -o out.txt myscript.py I get an error message output from the Python pickle module.

  File "<string>", line 1, in <module> File "c:\python27\lib\multiprocessing\forking.py", line 374, in main self = load(from_parent) File "c:\python27\lib\pickle.py", line 1378, in load return Unpickler(file).load() File "c:\python27\lib\pickle.py", line 858, in load dispatch[key](self) File "c:\python27\lib\pickle.py", line 880, in load_eof raise EOFError EOFError 

The main template in the code

 class MyProcess(multiprocessing.Process): def __init__(self, in_queue, msg_queue): multiprocessing.Process.__init__(self) self.in_queue = in_queue self.ext_msg_queue = msg_queue self.name == multiprocessing.current_process().name def run(self): ## Do Stuff with the queued items 

This usually loads from the GUI, but for testing purposes, I configure it as follows.

 if __name__ == '__main__': queue = multiprocessing.Queue() meg_queue = multiprocessing.Queue() p = Grabber(queue) p.daemon = True p.start() time.sleep(20) p.join() 

But when I try to run the script, I get the above error message.

Is there any way about error?

Answer 1

In my understanding, the cProfile module cProfile not work well with multiprocessing on the command line. (See Python Multiprocessing Profiling .)

One way around this is to run the profiler from your code - in particular, you will need to configure a different profiler output file for each process in your pool. You can do this quite easily by referring to cProfile.runctx('a+b', globals(), locals(), 'profile-%s.out' % process_name) .

http://docs.python.org/2/library/profile.html#module-cProfile