Call system () from a multithreaded program

Question

We are working on a multi-threaded memory-consuming application written in C ++. We need to execute many shellscript / linux commands (and get the return code).

After reading this article, we clearly understood that it would be a bad idea to use system () in our context.

The solution will be to fork after starting the program and before creating any threads, but communication with this process can be difficult (socket, pipe?).

The second solution that we examined may consist of a dedicated python daemon (using xinetd?) That can handle our system calls.

Have you ever had this problem? How did you solve it?

Note: Here is a more complete article explaining this problem: http://developers.sun.com/solaris/articles/subprocess/subprocess.html They recommend using posix_spawn, which uses vfork () instead of fork () (used in system () )

Answer 1

The article you are referencing mainly talks about problems if you fork () and don't immediately follow it with exec * (). Since system () will typically be implemented by fork (), followed by exec (), most problems do not apply. However, one problem lies in closing file descriptors; Unless you have particular reason to do this, opening files with O_CLOEXEC by default is probably a good rule.

One of the problems with fork () + exec () for large applications that consume large amounts of memory is that if your OS is configured to prevent memory from being re-arranged, fork () may fail. One solution to this is to fork the process as an “external process handler” before you start allocating a lot of memory into your main process.

The best solution is if the required functionality is available as a library, first of all eliminating the need for a plug. This probably does not warm your heart in the short term.

Answer 2

You will have questions about how you should call an external program (fork / exec / wait, how else), but this is only one part of the problem. The real problem is planning this, I suppose you do not want to run too many external programs in parallel.

Not knowing how threading works in your system, I can warn you about two problems.

An important issue is load reduction by limiting the external command / script call. You can configure a parameter that indicates how many parallel external commands should be executed at the same time. Before calling an external command, you must increase the variable, which shows the number of active external processes; if it exceeds the limit parameter, sleep () some and try again. After the process is complete, reduce this variable. (Increase and decrease must be mutually labeled.)

Another problem is that you are using an external program that controls its lifetime. You must set a timeout for each external process and kill it if it freezes for a while. There must be a "timeout" stream (or should be the main stream) that controls the others.

Answer 3

How about this solution:

fork () at the very beginning of your program and make your child dedicated to launch and manage your external programs. Then the parent will start all its threads and execute the application logic, sending requests over the channel when it needs an external process.

This will circumvent threading issues while you run them.