Executing commands in the background with a TCL script and formatting output

Question

I have a tcl script that runs several shell commands sequentially.

Something like that:

abc.tcl

command 1 command 2 command 3 ... command n 

This script outputs the output of these commands to a text file in the following format:

 ### ### ### ### ### ### Command name ### ### ### ### ### ### Command Output ### ### ### ### ### ## 

I tried to get the script to work faster, but at the same time shell commands run in parallel, and not one at a time. By pushing them into the background (a & command). But I don’t understand how to keep the formatting of my output text file as it was before.

When I press commands in the background, I am forced to add their outputs to a temporary file, but these files just have the output of the commands in a dump together. It is difficult to distinguish between different outputs.

Is there a way to redirect the output of each command running in the background to a separate temp file (perhaps the name of the temporary file may contain the process identifier in the background). And as soon as all the commands are launched, can I output the outputs together in the correct format? Any ideas / suggestions on how I can do this.

Answer 1

If the commands do not have states that depend on each other, you can parallelize them. There are many ways to do this, but one of them is to use thread pooling (which requires threading Tcl, which is the norm for many platforms at present):

 package require Thread set pool [tpool::create -maxworkers 4] # The list of *scripts* to evaluate set tasks { {command 1} {command 2} ... {command n} } # Post the work items (scripts to run) foreach task $tasks { lappend jobs [tpool::post $pool $task] } # Wait for all the jobs to finish for {set running $jobs} {[llength $running]} {} { tpool::wait $pool $running running } # Get the results; you might want a different way to print the results... foreach task $tasks job $jobs { set jobResult [tpool::get $pool $job] puts "TASK: $task" puts "RESULT: $jobResult" } 

The main setting is the thread pool size, which by default is equal to limit 4. (Set it using the -maxworkers parameter to tpool::create , which I listed explicitly above.) The best value for choosing depends on how many processor cores you have and how many CPU load, each task is generated on average; you will need to measure and adjust ...

You can also use the -initcmd option to preload each workflow in the pool using the script of your choice. This is a good place to place your package require calls. Workers are all completely independent of each other and of the leading thread; they do not divide the state. You would get the same model if you ran each piece of code in a separate process (but then you ended up writing more code for coordination).

---

[EDIT]: Here is a version that will work with Tcl 8.4 and which uses subprocesses instead.

 namespace eval background {} proc background::task {script callback} { set f [open |[list [info nameofexecutable]] "r+"] fconfigure $f -buffering line puts $f [list set script $script] puts $f {fconfigure stdout -buffering line} puts $f {puts [list [catch $script msg] $msg]; exit} fileevent $f readable [list background::handle $f $script $callback] } proc background::handle {f script callback} { foreach {code msg} [read $f] break catch {close $f} uplevel "#0" $callback [list $script $code $msg] } proc accumulate {script code msg} { puts "#### COMMANDS\n$script" puts "#### CODE\n$code" puts "#### RESULT\n$msg" # Some simple code to collect the results if {[llength [lappend ::accumulator $msg]] == 3} { set ::done yes } } foreach task { {after 1000;subst hi1} {after 2000;subst hi2} {after 3000;subst hi3} } { background::task $task accumulate } puts "WAITING FOR TASKS..." vwait done 

Notes: Tcl commands are tasks that produce a result, but they should not print the result; fabric code (in background::task ) handles this. These are subprocesses; they don’t share anything with each other, so everything that you want them to do or set up should be sent as part of the task. A more sophisticated version might support a hot pool of subprocesses around and generally work very similar to a thread pool (given the subtle differences due to the fact that it is in the subprocess and not in the thread), but it was more code than I wanted to write here.

The result codes (i.e. exception codes) are 0 for "ok", 1 for "error" and other values ​​in less common cases. These are exactly the values ​​described on the Tcl 8.6 catch page; it is for you to interpret them correctly. (I suppose I should add code so that the contents of the ::errorInfo and ::errorCode returned in case of an error, but this makes the code more complicated ...)