Python should use streams

Question

I currently have several small applications <500 lines. I intend to eventually run them on a small LINUX ARM unit. Is it better to combine these applications and use streams or continue to use them as two separate applications?

These applications plus a very small website use a small sqlite database, although only one of the applications writes everything that it currently reads. Due to limitations in the target field, I am using Python 2.6.

I use SQLite to prevent data loss for several days of use. There is no direct interaction between the two applications, although it is possible to lock the database, especially during the maintenance of the data period. Stopping these issues is troubling and the footprint of the two applications, since the target devices are quite small.

Answer 1

Depending on whether you need data to exchange data and how sharing is done. In addition, in terms of speed, for a multiprocessor machine, streaming will not give you a big advantage over individual processes.

If sharing can be easily done through a flat file or database, just let them be separate and not complicate the streaming process.

Answer 2

To ensure performance, I suggest you use threads, the process consumes much more resources than threads, it will create and consume less memory faster (useful in the embedded environment), but, of course, you will have to deal with common multi-threading traps (access matching is allowed by locks, but locks can lead to locks ...)

If you plan to use many libraries that make low-level calls, possibly with a C extension that could not effectively filter the Global Interpreter Lock (GIL), then processes might be the best solution so your applications can run even if one of which GIL is blocked.

Answer 3

If you need to transfer data between them, you can use Queues and other mechanisms in the multiprocessing module.

It is often much easier to use multiprocessing rather than sharing memory or objects using the streaming module.

If you do not need to transfer data between your programs, just run them separately (and release file or database locks as soon as possible to reduce the conflict).

Answer 4

I decided to adopt a process, not a multi-threaded approach to solving this problem. The main factor in this decision is simplicity. The second factor is that while one of these applications will collect data, the other will communicate with the modem on a one-time basis (receiving calls). I do not control the calling application, but based on my research, it is potential for many to go wrong. There are several factors that can change the approach further along the line, first of all, the need for the interaction of two processes to prevent data competition in the database. Secondly, if problems (memory / disk space / processor) become a problem (due to the size of the device), one application should enable me to manage these problems. However, the data collection application is already threaded. This allows the parent thread to control the worker when exceptions occur because the device will not be in a managed environment.