How can I reduce the number of commands executed inside a bash script?

Question

Suppose I have a bash script that executes code like this:

for i in $LIST; do /path/to/my/script.sh $i & done 

As you can see, I am running these scripts in the background and letting the parent script execute as many commands as possible. The problem is that my system will eventually run out of memory, as these commands take about 15 or 20 seconds to start each instance.

I run one static script.sh file and pass a simple variable (i.e. client number) to the script. There are about 20,000 - 40,000 entries that I scroll through at any given time.

My question is how can I tell the system that only the X-number of instances of script.sh is launched immediately. If too many are running, I want to pause the script until the number of scripts is below the threshold and then continues.

Any ideas?

Answer 1

A bash is the only solution:

 MAXJOBS=<your-requested-max + 3> for i in $LIST; do /path/to/my/script.sh $i & while true; do NUMJOBS=`ps --ppid $$ -o pid= | wc | awk -F ' ' '{ print $1;}'` test $NUMJOBS -lt $MAXJOBS && break done 

Answer 2

Two tools can do it.

(note that I changed the file selection because I think you should prepare to handle strange file names, e.g. with spaces)

GNU xargs

 find -iname '*.txt' -print0 | xargs -0 -r -n1 -P4 /path/to/my/script.sh 

Works in parallel on 4 processors

Xjobs

 find -iname '*.txt' -print0 | xjobs -0 /path/to/my/script.sh 

Powered by so many processors that you have. Xjobs does a better job of sharing the output of different jobs than xargs does.

Add -j4 to run 4 jobs in parallel

Answer 3

One simple hack is to create a Makefile that executes each of the scripts and runs make -jX :

 all : $(LIST) % : /path/to/my/script.sh $^ $* 

A good side effect is that make will automatically detect when your script has changed, but for this you will need to replace the % pattern with the name of any output file your script generates for this input parameter (assuming it does). For instance:.

 out.%.txt: /path… 

Answer 4

You must use xargs with -P. Create a script structure as follows:

 echo "$LIST" | xargs -n1 -P $SIMULTANEOUS_JOBS /path/to/my/script.sh 

Where, of course, SIMULTANEOUS_JOBS - how many commands do you want to run right away.

Answer 5

You might be interested in the parallel command from the Joey Hess moreutils package . [*] Use would be

 parallel -j MAXJOBS /path/to/my/script.sh -- $LIST 

[*] Not to be confused with the more powerful, but difficult to use, GNU parallel .

Answer 6

GNU Parallel is designed for the following tasks:

 parallel /path/to/my/script.sh ::: $LIST 

This will run one script.sh on each core.

Watch the videos to learn more:

http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Answer 7

I always like to do a little recursion:

 #!/bin/bash max=3 procname="journal" calltask() { if [ "$(ps -ef | grep ${procname} | grep -v grep | wc -l)" -le "${max}" ]; then echo " starting new proc " calltask else echo "too many processes... going to sleep" sleep 5 calltask fi } calltask