Is BASH multi-threaded programming a generic method?

Question

Ok, I ran POV-Ray in all demos, but POV is still single-threaded and has not used more than one core. So, I started thinking about a solution in BASH.

I wrote a general function that takes a list of commands and runs them in the specified number of sub-shells. This really works, but I don't like the way it processes the following command in the thread-safe multi-process way :

• As an argument, a command file is required (1 per line),
• To get the "next" command, each process ("thread") will be:
  • Waits until it can create a lock file using: ln $ CMDFILE $ LOCKFILE
  • Read the command from the file,
  • Modifies $ CMDFILE by deleting the first line,
  • Removes $ LOCKFILE.

Is there a cleaner way to do this? I could not get the sub-shells to correctly read one line from FIFO.

---

By the way, the point is to improve what I can do on the BASH command line , and not find solutions not bash. I try to perform many complex tasks from the command line and want the tool to have another tool.

Meanwhile, here is a function that handles getting the next line from a file. As you can see, it modifies the file on disk every time it reads / deletes a line. This is what seems hacky, but I did not come up with anything better, since FIFO did not work without setvbuf () in BASH.

#
# Get/remove the first line from FILE, using LOCK as a semaphore (with
# short sleep for collisions).  Returns the text on standard output,
# returns zero on success, non-zero when file is empty.
#
parallel__nextLine() 
{
  local line rest file=$1 lock=$2

  # Wait for lock...
  until ln "${file}" "${lock}" 2>/dev/null
  do sleep 1
     [ -s "${file}" ] || return $?
  done

  # Open, read one "line" save "rest" back to the file:
  exec 3<"$file"
  read line <&3 ; rest=$(cat<&3)
  exec 3<&-

  # After last line, make sure file is empty:
  ( [ -z "$rest" ] || echo "$rest" ) > "${file}"

  # Remove lock and 'return' the line read:
  rm -f "${lock}"
  [ -n "$line" ] && echo "$line"
}

Answer 1

#adjust these as required
args_per_proc=1 #1 is fine for long running tasks
procs_in_parallel=4

xargs -n$args_per_proc -P$procs_in_parallel povray < list

, nproc, , . , -P

Answer 2

, .

python, , /.

Answer 3

, , IMO
, script , .

#!/bin/bash

pids=()
thread() {
  local this
  while [ ${#} -gt 6 ]; do
    this=${1}
    wait "$this"
    shift
  done
  pids=($1 $2 $3 $4 $5 $6)
}
for i in 1 2 3 4 5 6 7 8 9 10
do
  sleep 5 &
  pids=( ${pids[@]-} $(echo $!) )
  thread ${pids[@]}
done
for pid in ${pids[@]}
do
  wait "$pid"
done

seems to work just fine for what I am doing (handling parallel loading of several files at the same time) and does not allow it to break my server, while still making sure that all the files are downloaded before the script completes

Answer 4

I believe that you are actually handling processes, not threads. I would recommend looking for streaming support in another scripting language, for example perl, pythonor ruby.