Shared data using multiple processors in python

Question

I have an intensive cpu code that uses a heavy dictionary like data (about 250M data). I have a multi-core processor and you want to use it so that I can run more than one task at a time. The dictionary is mostly read-only and can be updated once a day.
How can I write this in python without duplicating a dictionary?
I understand that python threads do not use native threads and will not offer true concurrency. Can I use a multiprocessor module without transferring data between processes?

I came from the java world, and my requirement would be something like Java threads that can exchange data, run on multiple processors and offer synchronization primitives.

Answer 1

You can exchange read-only data between processes simply using fork(on Unix, without is simple on Windows), but this will not catch a “one-time change” (you will need to put an explicit way for each process to update its own copy). Native Python structures, such as dict, are simply not designed to work on arbitrary addresses in shared memory (you will have to code an option dictthat supports this in C), so they offer no consolation.

You can use Jython (or IronPython) to get a Python implementation with exactly the same multithreading capabilities as Java (or C #, respectively), including multiprocessing using multiple simultaneous threads.

Answer 2

shelve . , .

Answer 3

Take a look at this in stdlib: http://docs.python.org/library/multiprocessing.html There are many great features that let you easily share data structures between processes.