Python subprocess freezes

Question

Python version: 2.6.7 I have the following subprocess.call in a for loop that gets called 18 times, however the process hangs constantly in the 19th loop:

if config.get_bool_option(NAME, 'exclude_generated_code', True): for conf in desc.iter_configs(): for gen in desc.iter_generators(conf): generator.initialize_generated_path(gen, desc) for genpath in gen.generated_path: os.rename(cov_file, cov_file+'.temp') exclude = ['lcov'] exclude += ['-r', cov_file+'.temp', '"'+genpath+'/*"'] exclude += ['-o', cov_file] if verbose: Tracer.log.info("Running "+ ' '.join(exclude)) try: subprocess.call(' '.join(exclude), stdout=out, stderr=out, shell=True) except subprocess.CalledProcessError, e: if verbose: Tracer.log.info("TESTING: Got Exception \n") 

The console output is as follows:

 Running lcov -r /remote/XXXXXX/coverage.19.temp "/remote/XXXXXX/xml/2009a/generated/*" -o /remote/XXXXX/gcov/coverage.19 

Since I am not very familiar with python scripts, I just wandered around if I was doing something wrong here ... I suspect that somewhere is a dead end.

Can stdout, stderr = process.communicate() solve these problems?

Some kind of expert answer, in which cases the subprocess. Many thanks

Answer 1

When using a subprocess, I usually do something like this:

 is_running = lambda: my_process.poll() is None my_process = subprocess.Popen(' '.join(exclude), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True) # Grab all the output from stdout and stderr and log it while is_running(): rlist, wlist, xlist = select.select([my_process.stdout, my_process.stderr], [], [], 1) # Log stdout, but don't spam the log if my_process.stdout in rlist and verbose: # Adjust the number of bytes read however you like, 1024 seems to work # pretty well for me. Tracer.log.debug(my_process.stdout.read(1024)) # Log stderr, always if my_process.stderr in rlist: # Same as with stdout, adjust the bytes read as needed. Tracer.log.error(my_process.stderr.read(1024)) 

I saw how stdout files just unloaded empty lines in my logs in the past, so I log them at the debug level. This prints my logs during development, but never writes to production, so I can safely leave it in the debugging code without putting garbage in my logs.

Hope this helps expose where your program hangs and what causes it.