Using __getattr__ and matching expected behavior for subclasses

Question

I am the author - and currently almost certainly the only user, if on several projects - a simple database level (inspired by minimongo ) for MongoDB called kale . My current use of __getattr__ in the base class for models has led to some hard to reach errors.

The problem that I encountered was clearly formulated last June by David Halter on this site. The discussion is interesting, but no solutions have been proposed.

In short:

 >>> class A(object): ... @property ... def a(self): ... print "We're here -> attribute lookup found 'a' in one of the usual places!" ... raise AttributeError ... return "a" ... ... def __getattr__(self, name): ... print "We're here -> attribute lookup has not found the attribute in the usual places!" ... print('attr: ', name) ... return "not a" ... >>> print(A().a) We're here -> attribute lookup found 'a' in one of the usual places! We're here -> attribute lookup has not found the attribute in the usual places! ('attr: ', 'a') not a >>> 

Note that this conflicting behavior is not what I would expect from reading the official python documentation :

object.__getattr__(self, name)

Called when the attribute search did not find the attribute in (that is, it is not an instance attribute, and it is not found in the class tree for itself). name is the name of the attribute.

(It would be nice if they mentioned that AttributeError is a means by which the "attribute search" knows if the attribute was found in "normal places" or not. The clearing bracket seems to me incomplete at best.)

In practice, this caused problems with tracking errors caused by programming errors, where an AttributeError occurs in the @property descriptor.

 >>> class MessedAttrMesser(object): ... things = { ... 'one': 0, ... 'two': 1, ... } ... ... def __getattr__(self, attr): ... try: ... return self.things[attr] ... except KeyError as e: ... raise AttributeError(e) ... ... @property ... def get_thing_three(self): ... return self.three ... >>> >>> blah = MessedAttrMesser() >>> print(blah.one) 0 >>> print(blah.two) 1 >>> print(blah.get_thing_three) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 11, in __getattr__ AttributeError: 'get_thing_three' >>> 

In this case, it’s pretty obvious what happens by checking the class as a whole. However, if you rely on a message from the stack trace, AttributeError: 'get_thing_three' , this makes no sense, since obviously get_thing_three looks like a valid attribute as they appear.

kale goal is to provide a base class for building models. Thus, the code of the base model is hidden from the end programmer, and disguising the cause of errors like this is not ideal.

The end programmer (cough me) can use @property descriptors for their models, and their code should work and fail in the way they expected.

Question

How can I allow an AttributeError propagate through my base class that defined __getattr__ ?

Answer 1

Basically, you cannot, or at least not in a simple and reliable way. As you noted, AttributeError is a mechanism used by Python to determine if an attribute is found in regular places. Although the __getattr__ documentation does not mention this, the documentation for __getattribute__ , as described in this answer , makes it more clear that you are already associated with.

You can override __getattribute__ and catch AttributeError there, but if you caught it, you won’t have an obvious way to find out if it was a “real” error (this means the attribute was not really found) or an error caused by the user code during the search process . Theoretically, you can check the trace in search of specific things or make other other hacks to try to verify the existence of the attribute, but these approaches will be more fragile and dangerous than the existing behavior.

Another possibility is to write your own property-based descriptor, but catch attributes attributes and re-create them as something else. This, however, will require you to use this property replacement instead of the built-in property . In addition, this would mean that AttributeErrors attributes created from internal descriptor methods are not propagated as attribute attributes, but like something else (whatever you replace them with). Here is an example:

 class MyProp(property): def __get__(self, obj, cls): try: return super(MyProp, self).__get__(obj, cls) except AttributeError: raise ValueError, "Property raised AttributeError" class A(object): @MyProp def a(self): print "We're here -> attribute lookup found 'a' in one of the usual places!" raise AttributeError return "a" def __getattr__(self, name): print "We're here -> attribute lookup has not found the attribute in the usual places!" print('attr: ', name) return "not a" >>> A().a We're here -> attribute lookup found 'a' in one of the usual places! Traceback (most recent call last): File "<pyshell#8>", line 1, in <module> A().a File "<pyshell#6>", line 6, in __get__ raise ValueError, "Property raised AttributeError" ValueError: Property raised AttributeError 

Here AttributeErrors are replaced by ValueErrors. This may be good if you only want to make sure that the exception is "breaking out" of the attribute access mechanism and can be propagated to the next level. But if you have a complex exception code that expects to see AttributeError, it will skip this error because the type of exception has changed.

(Also, this example obviously deals only with property attributes, not setters, but it should be clear how to extend the idea.)

I suggest that as an extension of this solution, you could combine this MyProp idea with a custom __getattribute__ . Basically, you can define a custom exception class, say PropertyAttributeError , and have a remark of replacing AttributeError properties with PropertyAttributeError. Then in your custom __getattribute__ you can catch a PropertyAttributeError and raise it again as an AttributeError. Basically, MyProp and __getattribute__ can act as a "shunt" that bypasses normal Python processing, converting the error from AttributeError to another, and then converting it back to AttributeError again as soon as it is "safe" for this. However, I feel that this is not worth it, as __getattribute__ can have a significant impact on performance.

One small addition: a bug about this was raised in Python, and recently there has been information about possible solutions, so it may be possible to fix it in future versions.

Answer 2

What happens in your code:

first case with class A :

>>>print(A().a)

• create instance A
• access attribute 'A' in instance

now python, following its data model, is trying to find Aa using object.__getattribute__ (since you did not provide your own __getattribute__ )

but:

 @property def a(self): print "We're here -> attribute lookup found 'a' in one of the usual places!" raise AttributeError # <= an AttributeError is raised - now python resorts to '__getattr__' return "a" # <= this code is unreachable 

so since __getattribute__ search completed using AttributeError , it raises your __getattr__ :

  def __getattr__(self, name): ... print "We're here -> attribute lookup has not found the attribute in the usual places!" ... print('attr: ', name) ... return "not a" #it returns 'not a' 

what happens in the second code:

you access blah.get_thing_three on __getattribute__ . since get_thing_three fails (no three in things ) with the AttributeError attribute, now your __getattr__ tries to look for get_thing_three in things , which also fails - you get an exception for get_thing_three because it has a higher precedence.

what can you do:

you need to write custom __getattribute__ and __getattr__ both. but in most cases it won’t make you far, and other people using your code do not expect some user data protocols.

Well, I have some advice for you (I wrote a rude orgon Mongodba that I use internally): don't rely on your __getattr__ inside your document. use direct access to the document inside your class (I think this will not hurt encapsulation). here is my example:

 class Model(object): _document = { 'a' : 1, 'b' : 2 } def __getattr__(self, name): r"""syntactic sugar for those who are using this class externally. >>>foo = Model() >>>foo.a 1""" @property def ab_sum(self): try : return self._document[a] + self._document[b] except KeyError: raise #something that isn't AttributeError 

Answer 3

I hope I can still offer some more ideas. So far, nothing satisfies my requirements! It may not be possible, but at least I got a little closer:

 >>> class GetChecker(dict): ... def __getattr__(self, attr): ... try: ... return self[attr] ... except KeyError as e: ... if hasattr(getattr(type(self), attr), '__get__'): ... raise AttributeError('ooh, this is an tricky error.') ... else: ... raise AttributeError(e) ... ... @property ... def get_thing_three(self): ... return self.three ... >>> >>> blah = GetChecker({'one': 0}) >>> print(blah.one) 0 >>> print(blah.lalala) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 6, in __getattr__ AttributeError: type object 'GetChecker' has no attribute 'lalala' >>> print(blah.get_thing_three) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 7, in __getattr__ AttributeError: ooh, this is an tricky error. >>> 

At least I can provide an error message that tells me how to track the problem, instead of looking like a problem ...

I am not satisfied yet. I will gladly accept an answer that can do better!