Boost.MultiIndex: is there a way to share an object between two processes?

Question

I have a large Boost.MultiIndex array of about 10 GB. To reduce reading, I thought that there should be a way to store data in memory, and other client programs will be able to read and analyze them.

What is the right way to organize it?

The array looks like this:

struct particleID { int ID;// real ID for particle from Gadget2 file "ID" block unsigned int IDf;// postition in the file particleID(int id,const unsigned int idf):ID(id),IDf(idf){} bool operator<(const particleID& p)const { return ID<p.ID;} unsigned int getByGID()const {return (ID&0x0FFF);}; }; struct ID{}; struct IDf{}; struct IDg{}; typedef multi_index_container< particleID, indexed_by< ordered_unique< tag<IDf>, BOOST_MULTI_INDEX_MEMBER(particleID,unsigned int,IDf)>, ordered_non_unique< tag<ID>,BOOST_MULTI_INDEX_MEMBER(particleID,int,ID)>, ordered_non_unique< tag<IDg>,BOOST_MULTI_INDEX_CONST_MEM_FUN(particleID,unsigned int,getByGID)> > > particlesID_set; 

Any ideas are welcome.

Regards, Arman.

EDIT: RAM and number of cores are not limited. I currently have 16Gb and 8cores.

Update

The same question I asked on the Boost.Users forum was answered by Joaquín M López Muñoz (developer of Boost.MultiIndex). Aveter Yes . You can split multi_index between processes using Boost.Interprocess. More details can be found in this link.

Answer 1

Have you looked at Boost.Interprocess ?

Answer 2

Have you thought to cut it to pieces.

Parallel access is difficult. It’s hard to get the right, it’s hard to maintain, it’s hard to reason.

On the other hand, 10 GB is very large, and I wondered if you can combine your data. Keep the same index structure, but send it to 10 (or more) independent objects depending on some conditions (for example, a large id).

Thus, you can naturally process each piece separately from the other, without having any problems with simultaneous access.