Why is Python “proactive” hanging trying to compute a very large number?

Question

I asked this question before killing a process that uses too much memory, and I have most of the solution developed.

However, there is one problem: calculating massive numbers does not seem to be affected by the method I'm trying to use. This code below is for setting a 10 second processor time per process.

import resource import os import signal def timeRanOut(n, stack): raise SystemExit('ran out of time!') signal.signal(signal.SIGXCPU, timeRanOut) soft,hard = resource.getrlimit(resource.RLIMIT_CPU) print(soft,hard) resource.setrlimit(resource.RLIMIT_CPU, (10, 100)) y = 10**(10**10) 

What I expect to see when running this script (on a Unix machine) is this:

 -1 -1 ran out of time! 

Instead, I get no output. The only way to get the result is Ctrl + C , and I get this if I Ctrl + C after 10 seconds:

 ^C-1 -1 ran out of time! CPU time limit exceeded 

If I Ctrl + C to 10 seconds, then I have to do it twice, and the console output looks like this:

 ^C-1 -1 ^CTraceback (most recent call last): File "procLimitTest.py", line 18, in <module> y = 10**(10**10) KeyboardInterrupt 

In the course of experiments and attempts to understand this, I also put time.sleep(2) between calculating print and a large number. This seems to have no effect. If I change y = 10**(10**10) to y = 10**10 , then the print and sleep statements will work as expected. Adding flush=True to the print statement or sys.stdout.flush() after the print statement also fails.

Why can't I limit the processor time to calculate a very large number? How can I fix or at least mitigate this?

---

Additional Information:

Python version: 3.3.5 (default, Jul 22 2014, 18:16:02) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)]

Linux information: Linux web455.webfaction.com 2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Answer 1

TL; DR: Python pre-computes the constants in the code. If any very large number is calculated with at least one intermediate step, the process will be limited by processor time.

---

It took quite a bit of searching, but I found evidence that Python 3 does the precompression of the constant literals that it finds in the code before evaluating anything. One of them is the web page: Peephole Optimizer for Python . I have cited some of them below.

ConstantExpressionEvaluator

This class precomputes a series of constant expressions and stores them in a list of function constants, including obvious binary and unary operations and tuples consisting of simple constants. Of particular note is the fact that complex literals are not represented by the compiler as constants, but as expressions, therefore 2 + 3j appears as

LOAD_CONST n (2) LOAD_CONST m (3j) BINARY_ADD

This class converts them to

LOAD_CONST q (2+3j)

which can lead to a pretty big performance improvement for code that uses complex constants.

The fact that 2+3j used as an example very strongly suggests that not only small constants are precomputed and cached, but also any constant literals in the code. I also found this comment in another overflow question ( Are persistent computations cached in Python? ):

Note that for Python 3, the peephole optimizer will precede the 1/3 constant. (Specific CPython, of course.) - Mark Dickinson October 7th at 7:40 pm

This is confirmed by the fact that the replacement

 y = 10**(10**10) 

hangs with this too, although I never call a function!

 def f(): y = 10**(10**10) 

---

Good news

Luckily for me, I don't have such giant literals in my code. Any calculation of such constants will occur later, which may be limited by the time limit of the CPU. I changed

 y = 10**(10**10) 

 x = 10 print(x) y = 10**x print(y) z = 10**y print(z) 

and got this way out at will!

 -1 -1 10 10000000000 ran out of time! 

Moral of the story: Limiting a process by processor time or memory consumption (or some other method) will work if there is no large literal constant in the code that Python is trying to precompute.

Answer 2

Use the function.

Python seems to be trying to precompute integer literals (I only have empirical data, if anyone has a source, please let me know). This will usually be a useful optimization, since the vast majority of literals in scripts are probably small enough not to cause noticeable delays in the preliminary calculation. To get around this, you need to make your literal the result of non-constant calculations, such as calling a function with parameters.

Example:

 import resource import os import signal def timeRanOut(n, stack): raise SystemExit('ran out of time!') signal.signal(signal.SIGXCPU, timeRanOut) soft,hard = resource.getrlimit(resource.RLIMIT_CPU) print(soft,hard) resource.setrlimit(resource.RLIMIT_CPU, (10, 100)) f = lambda x=10:x**(x**x) y = f() 

This gives the expected result:

 xubuntu@xubuntu-VirtualBox:~/Desktop$ time python3 hang.py -1 -1 ran out of time! real 0m10.027s user 0m10.005s sys 0m0.016s