Pop-up Performance in C

Question

I am developing a program that I plan to implement in C, and I have a question about the best way (in terms of performance) of invoking external programs. A user will provide my program with a file name, and then my program will run another program with this file as input. Then my program will process the output of another program.

My typical approach would be to redirect the output of another program to a file, and then my program would read that file when it is done. However, I understand that I / O operations are quite expensive, and I would like to make this program as efficient as possible.

I looked a little and I found the popen command to run system commands and capture output. How does the performance of this approach compare with the performance of the approach I described? popen just write the external output of the program to a temporary file or does it save the program in memory?

Alternatively, is there another way to do this in order to provide better performance?

Answer 1

On Unix systems, popen will transfer data through a pipe in memory. Assuming that the data has not changed, it will not get to disk. This should give you the highest possible performance than you can get without changing the called program.

Answer 2

popen does pretty much what you ask for: it executes the pipe-fork-exec idiom and gives you a pointer to a file that you can read and write.

However, there is a limit on the size of the channel buffer (~ 4K iirc), and if you read fast enough, another process may be blocked.

Do you have access to shared memory as a mount point? [Linux systems have a mount point / dev / shm]

Answer 3

1) popen to save the program output in memory. It actually uses channels to transfer data between processes.

2) popen looks like IMHO as the best option for performance.

It also has an advantage over files with reduced latency. That is, your program will be able to get another version of the program "on the fly" until it is released. If this output is large, then you do not need to wait until another program finishes to begin processing its output.

Answer 4

The problem with redirecting your subcommand to a file is that it is potentially unsafe, and the popen connection cannot be intercepted by another process. In addition, you must ensure that the file name is unique if you use multiple instances of your main program (and therefore your subcommand). The popen solution does not suffer from this.

The performance of popen just fine until you read / write single-byte snippets. Always read / write 512 (e.g., 4096). But this also applies to file operations. popen connects your process and the child process through channels, so if you do not read, the pipe fills up and the child cannot write and vice versa. Thus, all exchanged data is stored in memory, but these are just small amounts.

Answer 5

(Assume Unix or Linux)

Writing to a temporary file can be slow if the file is on a slow disk. It also means that all output will be installed on disk.

popen connects to another program using the channel, which means that the output will be sent to your program in stages. As it is generated, it is copied to your chunk-by-chunk program.