How can I profile a multithreaded program in Python?

Question

I am essentially developing a multi-threaded module in Python, and I would like to know where it spends its time. cProfile only seems to profile the main thread. Is there a way to profile all the threads involved in the calculation?

Answer 1

See yappi (another Python profiler).

Answer 2

Instead of running a single cProfile , you can run a separate instance of cProfile in each thread, and then combine the statistics. Stats.add() does this automatically.

Answer 3

If you do the extra work, you can write your own profiling class that implements profile(self, frame, event, arg) . It is called whenever a function is called, and you can pretty easily set up a structure to collect statistics.

Then you can use threading.setprofile to register this function in each thread. When a function is called, you can use threading.currentThread() to see what it works on. More information (and a ready-to-use recipe) can be found here:

http://code.activestate.com/recipes/465831/

http://docs.python.org/library/threading.html#threading.setprofile

Answer 4

Given that the main functions of your different threads are different, you can use the very useful profile_func() decorator from here .

Answer 5

I don’t know any profiling application that supports such a thing for python, but you could write a Trace class that writes log files, where you put information about when the operation started and when it ended, and how much time, which it consumed.

This is a simple and quick solution to your problem.