Python database remote access

Question

I have a large data set (185 GB) on which I plan to execute some machine learning algorithms. Data is located on a local computer with limited computational authority. I have access to a remote cluster where I can execute my expensive algorithms. It has 1 TB of memory and is pretty fast. But for some reason, I only have 2 GB (!) Of disk storage on the remote server.

I can connect to the cluster via SSH, is there any way in python so that I can load the database into RAM via SSH?

Any general tips on how to solve this problem are greatly appreciated.

Answer 1

You can use paramiko so that you can connect to SSH with Python. Then you can run commands that output your data and read it from the stream. This will work better than copying files because it does not require copying data to disk. If the data is in files, you can simply use paramiko for the catfiles and read the data from the stream.