A barrier in bash, can this be done easily?

Question

Say I have a bash script that runs three scripts in parallel

./script1 & ./script2 & ./script3 & 

Now let's say that. / script 4 depends on scripts1, script2 and script3. How can I make him wait for them by running all three scripts at the same time?

Answer 1

You can use the wait built-in command available in Bash and in some other shells.
^{(see equivalent WAITFOR command on Windows)}

wait documentation

Wait for the completion of each specified process and return its completion status.

 Syntax wait [n ...] Key n A process ID or a job specification 

Each n can be a process identifier or job specification; if work specification is given, all processes in this working pipeline were waiting.

If n is not given, all currently active child processes are waiting for, and the return status is zero.

If n indicates a non-existent process or job, status 127 returned. Otherwise, the return status is the exit status of the last process or expected job.

A simple solution

Below, wait waits indefinitely to complete all active active processes at the moment (i.e., in this case, three scripts).

 ./script1 & ./script2 & ./script3 & wait # waits for all child processes ./script4 

Store PID in local shell variables

 ./script1 & pid1=$! ./script2 & pid2=$! ./script3 & pid3=$! wait $pid1 $pid2 $pid3 # waits for 3 PIDs ./script4 

Save PID in temporary files

 ./script1 & echo $! >1.pid ./script2 & echo $! >2.pid ./script3 & echo $! >3.pid wait $(<1.pid) $(<2.pid) $(<3.pid) rm 1.pid 2.pid 3.pid # clean up ./script4 

This last solution pollutes the current directory with three files ( 1.pid , 2.pid and 3.pid ). One of these files may be corrupted before the wait call. Moreover, these files can be left in the file system in case of failure.

Answer 2

On the bash man page:

 wait [n ...] Wait for each specified process and return its termination status. Each `n` may be a process ID or a job specification.... If `n` is not given, all currently active child processes are waited for, and the return status is zero. 

---

The simplest implementation may be for your last script to run the rest. Thus, it is easy to store their PID and pass them to wait .

Answer 3

I quickly took off many years ago, but now I need nested parallelism. Here is what I came up with:

 # Run each supplied argument as a bash command, inheriting calling environment. # bash_parallel can be nested, though escaping quotes can be tricky -- define helper function for such cases. # Example: bash_parallel "sleep 10" "ls -altrc" function bash_parallel { ( i=0 unset BASH_PARALLEL_PIDS # Do not inherit BASH_PARALLEL_PIDS from parent bash_parallel (if any) for cmd in " $@ " do ($cmd) & # In subshell, so sibling bash_parallel wont interfere BASH_PARALLEL_PIDS[$i]=$! echo "bash_parallel started PID ${BASH_PARALLEL_PIDS[$i]}: $cmd" i=$(($i + 1)) done echo "bash_parallel waiting for PIDs: ${BASH_PARALLEL_PIDS[@]}" wait ${BASH_PARALLEL_PIDS[@]} ) # In subshell, so ctrl-c will kill still-running children. } 

Using:

 eisbaw@leno :~$ time (bash_parallel "sleep 10" "sleep 5") bash_parallel started PID 30183: sleep 10 bash_parallel started PID 30184: sleep 5 bash_parallel waiting for PIDs: 30183 30184 real 0m10.007s user 0m0.000s sys 0m0.004s