Difference between memcache and python dictionary

Question

In my current project, I use Memcache to store key-value pairs, but since communication is through a socket between my process and Memcache, which causes huge delays. We went with memcache because we had a requirement to store a large number of key-value pairs. But now I want to store the dictionary as a global data structure in my process. This is a good thing? Since the dictionary will be stored in the address space of processes. Suggestions, please ....

Answer 1

A common reason for using memcached is that you would like to distribute the cache between multiple computers, bearing in mind both the availability of data on all computers and the use of all computers. If these requirements are not applicable to you, and you only need cached data on one machine, then memcached does not offer you so much. In this case, moving the dictionary into your local process might be a good idea.

Answer 2

I wrote a detailed answer to this memcached "about" page. I drew and that’s it.

In short: if you have more than one process, the dictionary will not help. If you have more than one process / computer, you are going to burn tons of memory that could be used many times to save a lot of money and get more more.

Answer 3

If your data is not that big, you can simply compile your python dictionary into files using cPickle.dump or marshal.dump and reload it from the file using cPickle.load or marshal.load, and if you need to worry about disk space, you can use bz2 or gzip to compress / decompress while reading / overwriting the file.