Bash: limiting subshells in a for loop with a file list

Question

I tried to get a for loop to run multiple commands at the same time and tried to do this through subshells. Ive managed to combine the script below to check and it seems to be working fine.

#!/bin/bash
for i in {1..255}; do
  (
    #commands
  )&

done
wait

The only problem is that my actual loop will be for me in the * files, and then it just works, I guess, because it launched too many subshells to process. Therefore I added

#!/bin/bash
for i in files*; do
  (
    #commands
  )&
if (( $i % 10 == 0 )); then wait; fi
done
wait

which is now failing. Does anyone know about this? Either using another command to limit the number of subshells or provide a number for $ i?

Greetings

Answer 1

jobs. :.

wc -w <<<$(jobs -p)

, :

#!/bin/bash
for i in files*; do
  (
    #commands
  )&
  if (( $(wc -w <<<$(jobs -p)) % 10 == 0 )); then wait; fi
done
wait

---

@chepner :

bash 4.3 wait -n , - ,

Answer 2

xargs/

, concurrency:

printf '%s\0' files* | xargs -0 -P6 -n1 yourScript

-P6 - , xargs. 10, .

xargs, . , GNU Parallel.

: ?

files=( files* )
for i in "${!files[@]}"; do
    commands "${files[i]}" &
    (( i % 10 )) || wait
done

( , , ).

:

simultaneous() {
    while [[ $1 ]]; do
        for i in {1..11}; do
            [[ ${@:i:1} ]] || break
            commands "${@:i:1}" &
        done
        shift 10 || shift "$#"
        wait
    done
}
simultaneous files*

Answer 3

#!/bin/bash
for f in files*; do
  (
    #commands
  )&
  (( i++ % 10 == 0 )) && wait
done
wait

i, . reset , i %10 0 = 10, 20, 30 ..

Answer 4

Bash ≥4.3, wait -n:

#!/bin/bash

max_nb_jobs=10

for i in file*; do
    # Wait until there are less than max_nb_jobs jobs running
    while mapfile -t < <(jobs -pr) && ((${#MAPFILE[@]}>=max_nb_jobs)); do
        wait -n
    done
    {
        # Your commands here: no useless subshells! use grouping instead
    } &
done
wait

---

wait -n, - :

#!/bin/bash

set -m

max_nb_jobs=10

sleep_jobs() {
   # This function sleeps until there are less than $1 jobs running
   local n=$1
   while mapfile -t < <(jobs -pr) && ((${#MAPFILE[@]}>=n)); do
      coproc read
      trap "echo >&${COPROC[1]}; trap '' SIGCHLD" SIGCHLD
      [[ $COPROC_PID ]] && wait $COPROC_PID
   done
}

for i in files*; do
    # Wait until there are less than 10 jobs running
    sleep_jobs "$max_nb_jobs"
    {
        # Your commands here: no useless subshells! use grouping instead
    } &
done
wait

---

The advantage of this kind is that we make no assumptions about the time taken to complete the work. A new job starts as soon as there is room for it. In addition, all this is pure Bash, therefore it does not rely on external tools and (perhaps more importantly), you can use the Bash environment (variables, functions, etc.) without exporting them (arrays cannot be easily exported like this, what could be a huge pro).