Python did not catch MemoryError

Question

I wrapped code that could end without using a try / except block. However, although a MemoryError is thrown, it is not caught.

I have the following code:

while True: try: self.create_indexed_vocab( vocab ) self.reset_weights() break; except MemoryError: # Stuff to reduce size of vocabulary self.vocab, self.index2word = None, None self.syn0, self.syn1 = None, None self.min_count += 1 logger.info( ...format string here... ) 

I get the following Traceback:

 File "./make_model_tagged_wmt11.py", line 39, in <module> model.build_vocab(sentences) File "/root/CustomCompiledSoftware/gensim/gensim/models/word2vec.py", line 236, in build_vocab self.reset_weights() File "/root/CustomCompiledSoftware/gensim/gensim/models/word2vec.py", line 347, in reset_weights self.syn0 += (random.rand(len(self.vocab), self.layer1_size) - 0.5) / self.layer1_size File "mtrand.pyx", line 1044, in mtrand.RandomState.rand (numpy/random/mtrand/mtrand.c:6523) File "mtrand.pyx", line 760, in mtrand.RandomState.random_sample (numpy/random/mtrand/mtrand.c:5713) File "mtrand.pyx", line 137, in mtrand.cont0_array (numpy/random/mtrand/mtrand.c:1300) MemoryError 

I am running Python 2.7.3 under Ubuntu 12.04

The line reset_weights self.syn0 is exactly the line that I expect to raise an exception (it allocates a large array). The mysterious thing is that I cannot catch a memory error and do something that will make the array size smaller.

Are there any special circumstances that make it impossible to capture a MemoryError ?

Answer 1

Note that due to the underlying memory management architecture (Cs malloc () function), the interpreter may not always fully recover from this situation; nevertheless, it throws an exception, so a stack trace can be printed if the cause was a run-time program.

(see documents )

You can usually catch MemoryErrors. Not knowing what exactly happens with the release of MemoryError, I would suggest that you can't catch it when the shit really gets into the fan, and there is no more memory to handle it.

In addition, since you may not be able to really recover from it (see above), there will probably not be much point in catching it. You should really avoid running out of memory and limit the amount of memory your program uses, for example. only so that the list has a limited size.