Best multithreaded use of the Python.Popen & communication () subprocess?

Question

I am running several commands that can run for some time in parallel on a Linux machine running Python 2.6.

So, I used the method subprocess.Popenand process.communicate()to parallelize the execution team mulitple groups and capture the output immediately after.

def run_commands(commands, print_lock):
    # this part runs in parallel.
    outputs = []
    for command in commands:
        proc = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
        output, unused_err = proc.communicate()  # buffers the output
        retcode = proc.poll()                    # ensures subprocess termination
        outputs.append(output)
    with print_lock: # print them at once (synchronized)
        for output in outputs:
            for line in output.splitlines():
                print(line)

In another place it is called like this:

processes = []
print_lock = Lock()
for ...:
    commands = ...  # a group of commands is generated, which takes some time.
    processes.append(Thread(target=run_commands, args=(commands, print_lock)))
    processes[-1].start()
for p in processes: p.join()
print('done.')

Expected result: each output of a group of commands is displayed simultaneously, and their execution is performed in parallel.

(, , , - ), , , - "" "". ( reset shell, .)

'\r', . , splitlines(), , repr(), .

, Popen communicate() stdout/stderr. check_output shortcut Python 2.7, . , , , , .

Popen communicate() ?

Answer 1

, . . .

http://bitbucket.org/daybreaker/kaist-cs443/src/247f9ecf3cee/tools/manage.py

, Python.

Answer 2

, , run_commands, , , , . , , , ?

Answer 3

In your code example, I noticed your use:

for line in output.splitlines(): 

partially solve the " / r " problem ; using

for line in output.splitlines(True): 

would be helpful.