CompletedProcess from subprocess.run () does not return a string

Question

According to the Python 3.5 docs , subprocess.run () returns a CompletedProcess object with a stdout element that contains "a sequence of bytes, or a string if run () was called with universal_newlines = True." I see only a sequence of bytes, and not the line that I expected (hoped) would be equivalent to a text string. For instance,

import pprint import subprocess my_data = "" line_count = 0 proc = subprocess.run( args = [ 'cat', 'input.txt' ], universal_newlines = True, stdout = subprocess.PIPE) for text_line in proc.stdout: my_data += text_line line_count += 1 word_file = open('output.txt', 'w') pprint.pprint(my_data, word_file) pprint.pprint(line_count, word_file) 

Note: this uses a new feature in Python 3.5 that will not work in previous versions.

Do I need to create my own string buffering logic, or is there a way to get Python to do this for me?

Answer 1

proc.stdout already a string in your case, run print(type(proc.stdout)) to make sure. It contains all the subprocess output - subprocess.run() not returned until the child process is dead.

for text_line in proc.stdout: false: for char in text_string lists characters (Unicode code pages) in Python, not in strings. To get the lines, call:

 lines = result.stdout.splitlines() 

The result may differ from .split('\n') if there are Unicode lines in the line.

If you want to read the output of line by line (to avoid running out of memory for long processes):

 from subrocess import Popen, PIPE with Popen(command, stdout=PIPE, universal_newlines=True) as process: for line in process.stdout: do_something_with(line) 

Note: process.stdout is a file-like object in this case. Popen() does not Popen() for the process to complete - Popen() returns immediately after starting the child process. process is a subprocess.Popen instance, not a CompletedProcess here.

If you only need to count the number of lines (terminated by b'\n' ), for example wc -l :

 from functools import partial with Popen(command, stdout=PIPE) as process: read_chunk = partial(process.stdout.read, 1 << 13) line_count = sum(chunk.count(b'\n') for chunk in iter(read_chunk, b'')) 

See Why are reading lines from stdin much slower in C ++ than Python?

Answer 2

if you need to have STDOUT lines in an array to manipulate them better, you just skip the output separation by the "Universal new line" delimiters

 nmap_out = subprocess.run(args = ['nmap', '-T4', '-A', '192.168.1.128'], universal_newlines = True, stdout = subprocess.PIPE) nmap_lines = nmap_out.stdout.splitlines() print(nmap_lines) 

:

 ['Starting Nmap 7.01 ( https://nmap.org ) at 2016-02-28 12:24 CET', 'Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn', 'Nmap done: 1 IP address (0 hosts up) scanned in 2.37 seconds'] 

Answer 3

You see the line, compare:

 import subprocess proc = subprocess.run( args = [ 'cat', 'input.txt' ], universal_newlines = False, stdout = subprocess.PIPE) print (type(proc.stdout)) 

class 'bytes'

run popen.communicate calls

communication () returns a tuple (stdout_data, stderr_data). Data will be bytes or, if universal_newlines is True, strings.

See here for a more detailed explanation and other shell interactions.