Bash script to run a constant number of jobs in the background

Question

I need a bash script to do several jobs in the background, three jobs at a time.

I know that I can do this as follows, and to illustrate, I assume that the number of tasks is 6:

./j1 &
./j2 &
./j3 &
wait
./j4 &
./j5 &
./j6 &
wait

However, in this way, if, for example, j2 takes much longer to run j1 and j3, then I will stick to only one background job that has been running for a long time.

An alternative (which is exactly what I want) is that whenever one task is executed, bash should run the next task in the queue in order to support 3 tasks at any given time. Is it possible to write a bash script to implement this alternative, possibly using a loop? Please note that I need to run a lot more jobs, and I expect this alternative method to save me a lot of time.

Here is my script project, which I hope you can help me verify that it is correct and improve it since I am new to writing scripts in bash. The ideas in this script are taken and modified from here , here and here ):

for i in $(seq 6)
do
   # wait here if the number of jobs is 3 (or more)
   while (( (( $(jobs -p | wc -l) )) >= 3 )) 
   do 
      sleep 5      # check again after 5 seconds
   done

   jobs -x ./j$i &
done
wait

, , script . bash, - .

.

Answer 1

GNU xargs:

printf '%s\0' j{1..6} | xargs -0 -n1 -P3 sh -c './"$1"' _

---

bash (4.x) :

max_jobs=3; cur_jobs=0
for ((i=0; i<6; i++)); do
  # If true, wait until the next background job finishes to continue.
  ((cur_jobs >= max_jobs)) && wait -n
  # Increment the current number of jobs running.
  ./j"$i" & ((++cur_jobs))
done
wait

, , , - , , wait -n , . , - :

max_jobs=3
declare -A cur_jobs=( ) # build an associative array w/ PIDs of jobs we started
for ((i=0; i<6; i++)); do
  if (( ${#cur_jobs[@]} >= max_jobs )); then
    wait -n # wait for at least one job to exit
    # ...and then remove any jobs that aren't running from the table
    for pid in "${!cur_jobs[@]}"; do
      kill -0 "$pid" 2>/dev/null && unset cur_jobs[$pid]
    done
  fi
  ./j"$i" & cur_jobs[$!]=1
done
wait

... , , . xargs -P.:)

Answer 2

GNU Parallel:

parallel -j3 ::: ./j{1..6}

.. (, csh):

seq 6 | parallel -j3 ./j'{}'

, GNU Parallel, http://oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html , .

Answer 3

, .

usecase: "sleep 20" 30 , . script. , , " ?" " , ", while. , , , .5 .

Script output: , 30 "sleep 20" , , max = 30.

%_Host@User> ps -ef|grep 'sleep 20'|grep -v grep|wc -l
30
%_Host@User>

no. : Script "max", "max.txt" (max=$(cat max.txt)), while. , 45, 45 "sleep 20" , . Script "max.txt" .

%_Host@User> cat > max.txt
45
^C
%_Host@User> ps -ef|grep 'sleep 20'|grep -v grep|wc -l
45
%_Host@User>

Script:

#!/bin/bash
#---------------------------------------------------------------------#
proc='sleep 20' # Your process or script or anything..
max=$(cat max.txt)  # configure how many jobs do you want
curr=0
#---------------------------------------------------------------------#
while true
do
  curr=$(ps -ef|grep "$proc"|grep -v grep|wc -l); max=$(cat max.txt)
  while [[ $curr -lt $max ]]
        do
    ${proc} &        # Sending process to background.
    max=$(cat max.txt) # After sending one job, again calculate max and curr
    curr=$(ps -ef|grep "$proc"|grep -v grep|wc -l)
  done
  sleep .5    # sleep .5 seconds if reached max jobs.
done
#---------------------------------------------------------------------#

, .