Why the stunning overhead of [50X] contextlib and the With statement in Python and what to do with it

Question

In the process of searching for performance errors, I finally determined that the contextlib shell was the source of the problem. The overhead is quite staggering, and I did not expect this to slow down. The slowdown is in the 50X range, I cannot afford it in a loop. I would certainly appreciate the warning in the docs if it could slow down significantly.

It seems to have been known since 2010 https://gist.github.com/bdarnell/736778

It has a set of tests that you can try. Before starting, change fn to fn() in simple_catch() . Thank you DSM for this.

I am surprised that since then the situation has not improved. What can i do with this? I can go down to try / except, but I hope there are other ways to handle this.

Answer 1

Here are some new timings:

 import contextlib import timeit def work_pass(): pass def work_fail(): 1/0 def simple_catch(fn): try: fn() except Exception: pass @contextlib.contextmanager def catch_context(): try: yield except Exception: pass def with_catch(fn): with catch_context(): fn() class ManualCatchContext(object): def __enter__(self): pass def __exit__(self, exc_type, exc_val, exc_tb): return True def manual_with_catch(fn): with ManualCatchContext(): fn() preinstantiated_manual_catch_context = ManualCatchContext() def manual_with_catch_cache(fn): with preinstantiated_manual_catch_context: fn() setup = 'from __main__ import simple_catch, work_pass, work_fail, with_catch, manual_with_catch, manual_with_catch_cache' commands = [ 'simple_catch(work_pass)', 'simple_catch(work_fail)', 'with_catch(work_pass)', 'with_catch(work_fail)', 'manual_with_catch(work_pass)', 'manual_with_catch(work_fail)', 'manual_with_catch_cache(work_pass)', 'manual_with_catch_cache(work_fail)', ] for c in commands: print c, ': ', timeit.timeit(c, setup) 

I made simple_catch actually a function call, and I added two new tests.

Here is what I got:

 >>> python2 bench.py simple_catch(work_pass) : 0.413918972015 simple_catch(work_fail) : 3.16218209267 with_catch(work_pass) : 6.88726496696 with_catch(work_fail) : 11.8109841347 manual_with_catch(work_pass) : 1.60508012772 manual_with_catch(work_fail) : 4.03651213646 manual_with_catch_cache(work_pass) : 1.32663416862 manual_with_catch_cache(work_fail) : 3.82525682449 python2 p.py.py 33.06s user 0.00s system 99% cpu 33.099 total 

And for PyPy:

 >>> pypy bench.py simple_catch(work_pass) : 0.0104489326477 simple_catch(work_fail) : 0.0212869644165 with_catch(work_pass) : 0.362847089767 with_catch(work_fail) : 0.400238037109 manual_with_catch(work_pass) : 0.0223228931427 manual_with_catch(work_fail) : 0.0208241939545 manual_with_catch_cache(work_pass) : 0.0138869285583 manual_with_catch_cache(work_fail) : 0.0213649272919 

The overhead is much less than you claimed. In addition, the only utility PyPy, apparently, cannot delete with respect to try ... catch for the manual variant is to create an object, which in this case is trivially deleted.

---

Unfortunately, with too involved for good optimization with CPython , especially with regard to contextlib , which even PyPy makes optimization difficult. This is normal, because although creating an object + calling a function + creating a generator is expensive, it is cheap compared to what is usually done.

If you are sure that with causes most of your overhead, convert the context managers to cached instances, like mine. If this is too much overhead, you are likely to have a big problem with how your system is designed. Consider making the with scale bigger (this is usually not a good idea, but acceptable if necessary).

---

In addition, PyPy. DAT JIT will be fast.