Exchange between Python code and C ++ code (IPC)

Question

I have 2 code bases, one in python, one in C ++. I want to share real-time data between them. I am trying to evaluate which option is best for my specific use case:

• many small data updates from C ++ program to python program
• they both work on the same machine
• reliability is important
• low latency nice to have

I see several options:

• One process is written to a flat file, another process reads it. It is not scalable, slow and I / O error.
• One process writes to the database, another process reads it. This makes it more scalable, slightly less error prone, but still very slow.
• Insert my python program in C ++ one or vice versa. I rejected this decision because both codebases are quite complex, and I preferred to leave them separate for reasons of ease of maintenance.
• I use some sockets in both programs and send messages directly. This seems to be a reasonable approach, but it does not use the fact that they are on the same machine (it will be slightly optimized using the local host as the destination, but it will still be cumbersome).
• Using shared memory. So far, I think this is the most satisfactory solution I have found, but has the disadvantage that it is a little more difficult to implement.

Are there any other solutions that I should consider?

Answer 1

First of all, this question is very opinion based!

The cleanest way is to use them in one process and directly contact them. The only difficulty is implementing the correct APIs and C ++ -> Python calls. The disadvantages are the ease of maintenance, as you noted, and the potentially lower reliability (both in the event of a failure, and in most cases a problem) and lower flexibility (are you sure that you will not need to run them on different machines?). Extensibility is best because it is very easy to add more communication or modify existing ones. You can review the service point. Can you use a python application without mapping to C ++? If not for me, I would not have to worry about maintainability.

Then shared memory is the next choice with better maintainability, but with the same other drawbacks. Extensibility is a little worse, but still not so bad. It can be complicated, I don’t know Python support for working with shared memory, for C ++ you can take a look at Boost.Interprocess. The main question I need to check is the synchronization between processes.

Then, network communication. There are many options: from the simplest binary protocol implemented at the socket level to the higher levels mentioned in the comments. It depends on how complex your C ++ ↔ Python connection is and may be in the future. This approach may be more difficult to implement, third-party libraries may be required, but after that it will become extensible and flexible. Usually third-party libraries are based on code generation (Thrift, Protobuf), which does not simplify the build process.

I would not seriously consider the file system or database for this case.