Lazy-load variables using overloaded decorators

Question

I have a state object representing a system. The properties inside the state object are populated from [huge] text files. Since every access to the resource is carried out every time a state instance is created, it makes sense to lazily load them:

class State: def import_positions(self): self._positions = {} # Code which populates self._positions @property def positions(self): try: return self._positions except AttributeError: self.import_positions() return self._positions def import_forces(self): self._forces = {} # Code which populates self._forces @property def forces(self): try: return self._forces except AttributeError: self.import_forces() return self._forces 

There is a lot of repeating pattern. Moreover, sometimes import_abc can fill in several variables (i.e. import several variables from a small data file if it is already open).

It makes sense to overload @property so that it accepts the function "provide" this variable, namely:

 class State: def import_positions(self): self._positions = {} # Code which populates self._positions @lazyproperty(import_positions) def positions(self): pass def import_forces(self): self._forces = {} # Code which populates self._forces and self._strain @lazyproperty(import_forces) def forces(self): pass @lazyproperty(import_forces) def strain(self): pass 

However, I cannot find a way to trace exactly how the method is called in the @property decorator. So I don’t know how to approach @property overloading into my own @lazyproperty.

Any thoughts?

Answer 1

Perhaps you want something similar. This is a kind of simple memoization function combined with @property .

 def lazyproperty(func): values = {} def wrapper(self): if not self in values: values[self] = func(self) return values[self] wrapper.__name__ = func.__name__ return property(wrapper) class State: @lazyproperty def positions(self): print 'loading positions' return {1, 2, 3} s = State() print s.positions print s.positions 

What prints:

 loading positions set([1, 2, 3]) set([1, 2, 3]) 

Caution: entries in the value dictionary will not collect garbage, so it is not suitable for long programs. If the loaded value is unchanged for all classes, it can be stored on the function object itself for better speed and memory usage:

 try: return func.value except AttributeError: func.value = func(self) return func.value 

Answer 2

I think you can remove even more templates by writing your own descriptor class that will decorate the loader method. The idea is for the descriptor itself to encode the lazy loading logic, which means that the only thing you define in the real method is the loader itself (this is the only thing that apparently should really change for different values). Here is an example:

 class LazyDesc(object): def __init__(self, func): self.loader = func self.secretAttr = '_' + func.__name__ def __get__(self, obj, cls): try: return getattr(obj, self.secretAttr) except AttributeError: print("Lazily loading", self.secretAttr) self.loader(obj) return getattr(obj, self.secretAttr) class State(object): @LazyDesc def positions(self): self._positions = {'some': 'positions'} @LazyDesc def forces(self): self._forces = {'some': 'forces'} 

Then:

 >>> x = State() >>> x.forces Lazily loading _forces {'some': 'forces'} >>> x.forces {'some': 'forces'} >>> x.positions Lazily loading _positions {'some': 'positions'} >>> x.positions {'some': 'positions'} 

Please note that the lazy loading message was printed only on first access for each attribute. This version also automatically creates a "secret" attribute to store real data by adding an underscore to the method name (i.e., the data for positions stored in _positions . In this example, there is no setter, so you can 't do x.positions = blah ( although you can still change positions using x.positions['key'] = val ), but this approach can be extended to customize.

The best part about this approach is that your lazy logic is transparently encoded in the __get__ descriptor, which means that it easily generalizes to other types of templates that you might want to distract in this way.

Answer 3

However, I cannot find a way to determine exactly which method is called in the @property decorator.

property is actually a type (regardless of whether you use it with decorator syntax rather than orthogonal) that implements the descriptor protocol ( https://docs.python.org/2/howto/descriptor.html ). Too simplistic (I skipped deleter, doc and many other things ...) the pure-python implementation would look like this:

 class property(object): def __init__(self, fget=None, fset=None): self.fget = fget self.fset = fset def setter(self, func): self.fset = func return func def __get__(self, obj, type=None): return self.fget(obj) def __set__(self, obj, value): if self.fset: self.fset(obj, value) else: raise AttributeError("Attribute is read-only") 

Now overloading property not necessarily the easiest solution. There are actually quite a couple of existing implementations, including Django "cached_property" (cf http://ericplumb.com/blog/understanding-djangos-cached_property-decorator.html for more information about this) and pydanny " cached-property "( https://pypi.python.org/pypi/cached-property/0.1.5 )