How to implement a permanent list of "Python"?

Question

I am trying to make an object act like an inline list , except that its value is saved after changing it.

The implementation I came across includes a list in the PersistentList class. For each access to the method that can change the list, the wrapper delegates the wrapped list and stores it in the key database after it is called.

the code:

 class PersistentList(object): def __init__(self, key): self.key = key self._list = db.get(key, []) def __getattr__(self, name): attr = getattr(self._list, name) if attr: if attr in ('append', 'extend', 'insert', 'pop', 'remove', 'reverse', 'sort'): attr = self._autosave(attr) return attr raise AttributeError def _autosave(self, func): @wraps(func) def _(*args, **kwargs): ret = func(*args, **kwargs) self._save() return ret return _ def _save(self): db.set(self.key, self._list) 

There are several problems with this implementation:

• I have to decorate methods like append every time they are available, is there a better way to decorate several methods of some object?

• Operations like l += [1,2,3] do not work, because I do not have implemented the iadd method.

What can I do to simplify this?

Answer 1

I like @andrew cooke, but I see no reason why you cannot get directly from the list.

 class PersistentList(list): def __init__(self, *args, **kwargs): for attr in ('append', 'extend', 'insert', 'pop', 'remove', 'reverse', 'sort'): setattr(self, attr, self._autosave(getattr(self, attr)) list.__init__(self, *args, **kwargs) def _autosave(self, func): @wraps(func) def _func(*args, **kwargs): ret = func(*args, **kwargs) self._save() return ret return _func 

Answer 2

Here's a way to avoid having to sort each list method. This makes the PersistentList a context manager , so you can use

 with PersistentList('key', db) as persistent: do_stuff() 

syntax. Admittedly, this does not call the _save method after each list operation, only when you exit with-block . But I think that this gives you enough opportunities to save when you want to save, especially since the __exit__ method is guaranteed to be executed no matter how you leave with-block , including if this happens due to an exception.

You could be an advantage that _save not called after every list operation. Imagine adding 10,000 times to the list. How many individual calls to db.set (database?) Can be quite time consuming. I would be better off, at least in terms of performance, to make all the additions and save once.

---

 class PersistentList(list): def __init__(self, key, db): self.key = key self.extend(db.get(key, [])) def _save(self): # db.set(self.key, self) print('saving {x}'.format(x = self)) def __enter__(self): return self def __exit__(self,ext_type,exc_value,traceback): self._save() db = {} p = PersistentList('key', db) with p: p.append(1) p.append(2) with p: p.pop() p += [1,2,3] # saving [1, 2] # saving [1, 1, 2, 3] 

Answer 3

I know this is not beautiful or smart, but I will just write separate methods ...

 class PersistentList(object): ... def append(self, o): self._autosave() self._list.append(o) ...etc... 

Answer 4

Here is an answer that is very similar to @unutbu, but more general. It gives you a function that you can call to synchronize your object with the disk, and it works with other classes that are capable of scope, except list .

 with pickle_wrap(os.path.expanduser("~/Desktop/simple_list"), list) as (lst, lst_sync): lst.append("spam") lst_sync() lst.append("ham") print(str(lst)) # lst is synced one last time by __exit__ 

Here is the code that makes this possible:

 import contextlib, pickle, os, warnings def touch_new(filepath): "Will fail if file already exists, or if relevant directories don't already exist" # http://stackoverflow.com/a/1348073/2829764 os.close(os.open(filepath, os.O_WRONLY | os.O_CREAT | os.O_EXCL)) @contextlib.contextmanager def pickle_wrap(filepath, make_new, check_type=True): "Context manager that loads a file using pickle and then dumps it back out in __exit__" try: with open(filepath, "rb") as ifile: result = pickle.load(ifile) if check_type: new_instance = make_new() if new_instance.__class__ != result.__class__: # We don't even allow one class to be a subclass of the other raise TypeError(("Class {} of loaded file does not match class {} of " + "value returned by make_new()") .format(result.__class__, new_instance.__class__)) except IOError: touch_new(filepath) result = make_new() try: hash(result) except TypeError: pass else: warnings.warn("You probably don't want to use pickle_wrap on a hashable (and therefore likely immutable) type") def sync(): print("pickle_wrap syncing") with open(filepath, "wb") as ofile: pickle.dump(result, ofile) yield result, sync sync()