Running a limited number of parallel programs with a script

Question

Based on this solution I am trying to develop a script that would limit the number of processes running to 4. However, as an alternative, I want the jobs to be stored in an array that I reference the index. I wrote the following:

todo_array[1]="echo start1;sleep 3;echo done1" todo_array[2]="echo start2;sleep 3;echo done2" todo_array[3]="echo start3;sleep 3;echo done3" todo_array[4]="echo start4;sleep 3;echo done4" todo_array[5]="echo start5;sleep 3;echo done5" todo_array[6]="echo start6;sleep 3;echo done6" todo_array[7]="echo start7;sleep 3;echo done7" todo_array[8]="echo start8;sleep 3;echo done8" todo_array[9]="echo start9;sleep 3;echo done9" max_jobs=4 seq ${#todo_array[@]} | xargs -i --max-procs=$max_jobs bash -c $todo_array[{}] 

however, when executed, I get the empty output 9 newlinex. What am I doing wrong? Thanks

EDIT : I changed it to

 seq ${#todo_array[@]} | xargs -i --max-procs=$max_jobs bash -c "$todo_array[{}]" 

and I get the following weird conclusion:

 start1 start1 start1 start1 done1[2] done1[3] done1[1] done1[4] start1 start1 start1 start1 done1[5] done1[6] done1[7] done1[8] start1 done1[9] 

Answer 1

Try the following:

 #!/bin/bash todo_array[1]="echo start1;sleep 3;echo done1" todo_array[2]="echo start2;sleep 3;echo done2" todo_array[3]="echo start3;sleep 3;echo done3" todo_array[4]="echo start4;sleep 3;echo done4" todo_array[5]="echo start5;sleep 3;echo done5" todo_array[6]="echo start6;sleep 3;echo done6" todo_array[7]="echo start7;sleep 3;echo done7" todo_array[8]="echo start8;sleep 3;echo done8" todo_array[9]="echo start9;sleep 3;echo done9" max_jobs=4 for i in "${todo_array[@]}" do echo $i done | xargs -IX --max-procs=$max_jobs bash -c "X" 

Answer 2

If your question is not a training one, but simply “how to run a limited number of parallel programs,” my advice is not to use a script and reinvent the wheel, but use GNU Parallel , which is precisely created for this purpose.

Just create scripts with your commands and connect them in parallel, i.e.

 ./myScript.sh | parallel -j2 

In the example, the parallel will run two jobs in parallel, but it will collect the output from the scripts at the end, just as if you were actually running the scripts one after another. Of course, tasks must be independent; the question is not clear on this intention.

Answer 3

You may be interested in checking ppss . This is an easy to use powerful Threader for creating scripts / shells.