Iterate through a large 20+ gb file from the server using python

Question

Hi, I have about 16 20+ gb files on the server that I need to read certain records, I have code that reads the file in the correct order if I have one of the files stored on my computer.

f = open('biodayk1.H2009', 'rb') lbl = array.array('f') bio = 0 for day in range(iday): f.seek(nx*ny*km*bio*4, 1) lbl.read(f, nx*ny*km) #reads the desired ibio f.seek(nx*ny*km*(10 - bio)*4, 1) #skips the next ibios f.close() 

Now I need to read files from the server without downloading each file. I looked in paramiko and was able to connect to the server, but I'm not quite sure how to go through the file and just return what I want. If you need more information or if I need to answer any questions, please ask. Thanks in advance.

Answer 1

You are in pain. I recommend that you follow the rsync route and write a script that runs on a server that serves the bytes you are interested in. You can communicate with him through the text channel created by paramiko.

Answer 2

I would recommend execnet remotely run Python (local function or module).

No configuration required.