Using Python to copy a file via ssh to a remote bash script

Question

I am using subprocess.popen with shlex to invoke a remote bash script using ssh. This command works great on bash. But as soon as I try to translate it into python and shlex using subprocess.popen, it will fail.

Remote bash script:

#!/bin/bash tmp=""; while read -r line; do tmp="$tmp $line\n"; done; echo $tmp; 

BASH RESULT CMD (invoking the bash script console on the command line)

 $> ssh xxxx cat < /tmp/bef69a1d-e580-5780-8963-6a9b950e529f.txt " | /path/to/bash/script.sh;" Bar\n $> 

Python code

 import shlex import subprocess fn = '/tmp/bef69a1d-e580-5780-8963-6a9b950e529f.txt' s = """ ssh xxxx cat < {localfile} '| /path/to/bash/script.sh;' """.format(localfile=fn) print s lexer = shlex.shlex(s) lexer.quotes = "'" lexer.whitespace_split = True sbash = list(lexer) print sbash # print buildCmd proc=subprocess.Popen(sbash,stdout=subprocess.PIPE,stderr=subprocess.PIPE) out,err=proc.communicate() print "Out: " + out print "Err: " + err 

PYTHON script RESULT :

 $> python rt.py ssh xxxx cat < /tmp/bef69a1d-e580-5780-8963-6a9b950e529f.txt '| /path/to/bash/script.sh' ['ssh', 'xxxx', 'cat', '<', '/tmp/bef69a1d-e580-5780-8963-6a9b950e529f.txt', "'| /path/to/bash/script.sh'"] Out: Err: bash: /tmp/bef69a1d-e580-5780-8963-6a9b950e529f.txt: No such file or directory $> 

What am I missing?

Answer 1

The problem is that you are using shell redirection in a command, but when using a subprocess, the shell is not generated.

Consider the following (very simple) program:

 import sys print sys.argv 

Now, if we run it as if you were using ssh (suppose foofile.txt exists), we get:

 python argcheck.py ssh cat < foofile.txt " | /path/to/bash/script.sh;" ['argcheck.py', 'ssh', 'cat', ' | /path/to/bash/script.sh;'] 

Note that < foofile.txt never gets into python command line arguments. This is because the bash parser intercepts < and the file that appears after it and redirects the contents of this file to your stdin program. In other words, ssh reads the file from stdin. You want your file to be transferred to stdin from ssh , using python as well.

 s = """ ssh xxxx cat '| /path/to/bash/script.sh;' """ #<snip> proc=subprocess.Popen(sbash,stdout=subprocess.PIPE,stderr=subprocess.PIPE, stdin=subprocess.PIPE) out,err=proc.communicate(open(fn).read()) 

will work presumably.

---

The following works for me:

 import subprocess from subprocess import PIPE with open('foo.h') as f: p = subprocess.Popen(['ssh',' mgilson@XXXXX ','cat','| cat'],stdin=f,stdout=PIPE,stderr=PIPE) out,err = p.communicate() print out print '#'*80 print err 

And the equivalent command in bash :

 ssh mgilson@XXXXX cat < foo.h '| cat' 

where foo.h is the file on my local machine.

Answer 2

Why not give up an intermediary and use Paramiko ?