Perl - responsible forking

Question

I recently opened Perl and I am very in love. But one thing concerns me - if I just split processes from left to right, this will certainly cause some kind of problem. Is there a reasonable check that needs to be used to make sure that my small application does not eat up all the resources of my machine?

Take this sample code:

foreach my $command (@commands) { my $pid = fork(); if (!$defined $pid) { #Fork failed. Do something. } elsif ($pid == 0) { #This is the child. system($command); exit(0) } } while (wait() != -1) {} #wait() will be -1 when the last child exits. 

Thus, this will work fine and output the processing of each command. All this will happen in parallel, which is great if these teams are completely independent.

What if I suddenly have 5,000+ teams? It would be unreasonable to mindlessly abandon many processes. So which check should be implemented and how?

)

Answer 1

In addition, if you are worried that too many bloated processes are occurring at the same time, you can disable them.

Reset your own (using the fork queue for storage), or better yet, use the Parallel::ForkManager module, which allows you to restrict concurrent forks using the constructor parameter.

 use Parallel::ForkManager; $pm = new Parallel::ForkManager($MAX_PROCESSES); 

Please note that ForkManager will also take care of getting completed child processes for you through the provided "wait *" APIs

Answer 2

When the child leaves him, he will send back to the parent element SIG_CHLD . You will want to extract them as if you had not become a zombie in the process table before the final wait call at the end of your script.

Chapter 16 of O'Reilly Perl in Google Books contains tons of information about this. Basically, you need to increase the counter when you play the children, and decrease it when you reap them, rather than new forks that have a reasonable maximum of currently working children.

As for the "reasonable max" ... depends on the hardware and on what these forked processes do. There is no static answer to this question, except to say what you are doing, and look at the effect of performance on the machine. Preferably during business hours. Telling your system administrator what you are doing. He may even have some advice.

Answer 3

To make sure that you are not creating more processes than the system can process efficiently, you can use modules such as Parallel :: ForkManager

Answer 4

Spawning (forking), like many processes that your system can support, is called the bomb plug . Your system may freeze or crash, as its process table becomes saturated, and memory and processor are depleted. On Linux systems, the ulimit command should show the maximum number of user processes allowed. You can (should) correctly install it for your system. Viking creates processes exponentially. Thus, this snippet creates four processes that consume the processor endlessly until they are killed --- an unpleasant, tiny bomb:

 perl -e 'fork; fork; 1 while {}'