Distorted output of background jobs

Question

I am trying to collect line by line the output of several child processes in a bash script to redirect it to another process.

I did not find anything that would guarantee that the outputs of the subprocesses would not mix, but for me it is important that each output line corresponds to the output. The order between the outputs does not matter.

Here is an example of mixed / distorted output:

#!/bin/bash for i in {1..1000}; do ( { echo BEGIN; dmesg; echo END; } | tr -d '\n'; echo ) & done wait 

Launch:

 $ ./test_output.sh | perl -ne 'print "$1\n" if m/(.{1,20}BEGIN.{0,20})/' | head 0.000000] SRAT: PXMBEGIN[ 0.000000] Initi ME through PCIe PME BEGIN[ 0.000000] Initi ME through PCIe PME BEGIN[ 0.000000] Initi [ 0.209816] pci 0BEGIN[ 0.000000] Initi ciehp 0000:00:16.1:pBEGIN[ 0.000000] Initi CI: Updating contextBEGIN[ 0.000000] Initi l family 2[ 0.588BEGIN[ 0.000000] Initi ME through PCIe PME BEGIN[ 0.000000] Initi CI: Updating contextBEGIN[ 0.000000] Initi 3922 pages, LIFO batBEGIN[ 0.000000] Initi 

You can see several lines with mixed content.

Of course, without & everything is fine.

So, I have no choice but to redirect each child output to a file, and then after a big wait , cat all these files.

Doing the same with the GNU parallel does part of the job, but this is not an option in my environment.

The GNU parameter provides output from commands of the same output as you would receive if you ran the commands sequentially. This makes it possible to use the output from GNU in parallel as input for other programs.

So, parallel GNU will record each output of the task immediately after the completion of each task and take care not to mix the output. It's good. But I would also like to receive the exit of each work as soon as possible, that is, without waiting for the exit of the work. There is a "-u" switch, but then it will mix the output.

Do I need to play with fifo, choose or even write a perl script?

-

I think I found why / how / when the outputs mix in man 7 pipe

POSIX.1-2001 says that write (2) s less than PIPE_BUF bytes should be atomic: the output is written to the pipe as a continuous sequence. Records of more than PIPE_BUF bytes can be non-atomic: the kernel can alternate data with data written by other processes. POSIX.1-2001 requires that PIPE_BUF be at least 512 bytes. (On Linux, PIPE_BUF is 4096 bytes.)

Answer 1

Here is my first draw. This is a simple script that runs all the commands given on stdin in the background (not sure what I want) and collects the output from them, string strings.

 #!/usr/bin/env perl use strict; use warnings; use IO::Select; use POSIX qw(strftime); my $SELECT_TIMEOUT = 1; my $TAG_SEPARATOR = '|'; my $TAG_TIMESTAMP_FORMAT = '%Y-%m-%dT%H:%M:%S'; sub multiplex { my @commands = @_; my %tags = (); # fd -> cmd my $sel = IO::Select->new(); for my $cmd (@commands) { $cmd =~ s/^\s+|\s+$//g; my $fd; if (!open($fd, "-|", $cmd)) { warn "Cannot start '$cmd': $!"; next; } else { $tags{$fd} = $cmd; $sel->add($fd); } } while ($sel->handles > 0) { my @handles = $sel->can_read($SELECT_TIMEOUT); # maybe something went wrong if ( !@handles ) { for my $fd ($sel->has_exception($SELECT_TIMEOUT)) { $sel->remove($fd); } next; } my $now = strftime($TAG_TIMESTAMP_FORMAT, localtime(time())); for my $fd (@handles) { if (defined(my $line = <$fd>)) { if ($TAG_SEPARATOR) { $line = join($TAG_SEPARATOR, $now, $tags{$fd}, $line); } print $line; } else { # EOF $sel->remove($fd); } } } } multiplex(<STDIN>);