Accessing items from a dictionary efficiently using pickle in Python

Question

I have large dictionary matching keys (which are strings) for objects. I pickled this large dictionary, and at certain points I want to pull out only a few entries from it. A dictionary usually contains thousands of entries. When I load a dictionary using pickle as follows:

from cPickle import *
# my dictionary from pickle, containing thousands of entries
mydict = open(load('mypickle.pickle'))
# accessing only handful of entries here
for entry in relevant_entries:
  # find relevant entry
  value = mydict[entry]

I noticed that it might take up to 3-4 seconds to load the whole brine that I don't need, as I get access to a tiny subset of the words in the dictionary later (shown above).

How can I make it so that crumbling only downloads the entries that I have from the dictionary to make it faster?

Thank.

Answer 1

, , , - , . , .

, .

, . (ZODB, SQLite .), , .

Answer 2

Pickle (), . , , , , , . shelve, dbm (SQLite) .

Answer 3

You will need to have objects "Ghosts", Ie objects that are only placeholders and are loaded upon access. This is a difficult question, but it has been resolved. You have two options. You can use the persistence library from ZODB, which helps with this. Or you just start using ZODB directly; the problem is resolved.

http://www.zodb.org/