Sending a large file to PIPE input in python

Question

I have the following code:

sourcefile = open(filein, "r")
targetfile = open(pathout, "w")

content= sourcefile.read():

p = Popen([SCRIPT], stdout=targetfile, stdin=PIPE)
p.communicate(content)

sourcefile.close()
targetfile.close()

The data in the source file is quite large, so it takes a lot of memory / swap to store in the "contents". I tried to send the file directly to stdin using stdin = sourcefile, which works, except for the external script 'freezes', that is: it continues to wait for EOF. It may be an error in an external script, but it is no longer at my disposal.

Any tips on how to send a large file to an external script?

Answer 1

Replace p.communicate(content)with a loop that is read from sourcefileand written to p.stdinin blocks. If sourcefile- EOF, be sure to close p.stdin.

sourcefile = open(filein, "r")
targetfile = open(pathout, "w")

p = Popen([SCRIPT], stdout=targetfile, stdin=PIPE)
while True:
    data = sourcefile.read(1024)
    if len(data) == 0:
        break
    p.stdin.write(data)
sourcefile.close()
p.stdin.close()

p.wait()
targetfile.close()