Exception handling inside context managers

Question

I have some code where I try to access a resource, but sometimes it is unavailable and throws an exception. I tried to implement the retry mechanism using context managers , but I cannot handle the exception __enter__ by the caller in the __enter__ context for my context manager.

 class retry(object): def __init__(self, retries=0): self.retries = retries self.attempts = 0 def __enter__(self): for _ in range(self.retries): try: self.attempts += 1 return self except Exception as e: err = e def __exit__(self, exc_type, exc_val, traceback): print 'Attempts', self.attempts 

These are a few examples that just throw an exception (which I expected to handle)

 >>> with retry(retries=3): ... print ok ... Attempts 1 Traceback (most recent call last): File "<stdin>", line 2, in <module> NameError: name 'ok' is not defined >>> >>> with retry(retries=3): ... open('/file') ... Attempts 1 Traceback (most recent call last): File "<stdin>", line 2, in <module> IOError: [Errno 2] No such file or directory: '/file' 

Is there a way to catch these exceptions and handle them inside the context manager?

Answer 1

Quote __exit__ ,

If an exception is thrown and the method wants to suppress the exception (i.e., prevent its propagation), it should return the true value . Otherwise, the exception will usually be processed after exiting this method.

By default, if you do not return a value explicitly from a function, Python will return None , which is a false value. In your case, __exit__ returns None , and so the exception may go through __exit__ .

So return a true value like

 class retry(object): def __init__(self, retries=0): ... def __enter__(self): ... def __exit__(self, exc_type, exc_val, traceback): print 'Attempts', self.attempts print exc_type, exc_val return True # or any truthy value with retry(retries=3): print ok 

the conclusion will be

 Attempts 1 <type 'exceptions.NameError'> name 'ok' is not defined 

---

If you want to have a snooze function, you can implement this with a generator, like

 def retry(retries=3): left = {'retries': retries} def decorator(f): def inner(*args, **kwargs): while left['retries']: try: return f(*args, **kwargs) except NameError as e: print e left['retries'] -= 1 print "Retries Left", left['retries'] raise Exception("Retried {} times".format(retries)) return inner return decorator @retry(retries=3) def func(): print ok func() 

Answer 2

To handle the exception in the __enter__ method, the simplest (and less surprising) way would be to __enter__ with statement itself in a try-Except clause, and simply throw an exception -

But with blocks are definitely not designed for such work, that is, they are "repeating" themselves, and there is some misunderstanding:

 def __enter__(self): for _ in range(self.retries): try: self.attempts += 1 return self except Exception as e: err = e 

As soon as you return self , the context in which __enter__ is __enter__ no longer exists - if an error occurs in the with block, it will simply be passed to the __exit__ method in a natural way. And no, the __exit__ method cannot in any way force the thread to return to the beginning of the with block.

You probably want something more like this:

 class Retrier(object): max_retries = 3 def __init__(self, ...): self.retries = 0 self.acomplished = False def __enter__(self): return self def __exit__(self, exc, value, traceback): if not exc: self.acomplished = True return True self.retries += 1 if self.retries >= self.max_retries: return False return True .... x = Retrier() while not x.acomplished: with x: ... 

Answer 3

I think this is easy, and other people seem to overdo it. Just put the resource selection code in __enter__ and try to return, not self , but the extracted resource. In code:

 def __init__(self, retries): ... # for demo, let add a list to store the exceptions caught as well self.errors = [] def __enter__(self): for _ in range(self.retries): try: return resource # replace this with real code except Exception as e: self.attempts += 1 self.errors.append(e) # this needs to return True to suppress propagation, as others have said def __exit__(self, exc_type, exc_val, traceback): print 'Attempts', self.attempts for e in self.errors: print e # as demo, print them out for good measure! return True 

Now try:

 >>> with retry(retries=3) as resource: ... # if resource is successfully fetched, you can access it as `resource`; ... # if fetching failed, `resource` will be None ... print 'I get', resource I get None Attempts 3 name 'resource' is not defined name 'resource' is not defined name 'resource' is not defined