Multithreaded FTP upload. Is it possible?

Question

I need to upload several files from a directory to a server via FTP and SFTP. I solved this problem for SFTP using python, paramiko and threading. But I have a problem doing this for FTP. I tried using ftplib for python, but it seems that it doesn’t support streaming, and I upload all the files one by one, which is very slow.

I am wondering if it is possible to make multi-threaded downloads with the FTP protocol without creating separate connections / authorizations (it takes too much time)?

The solution can be in Python or PHP. Maybe CURL? I would be grateful for any ideas.

Answer 1

There is a problem creating an asynchronous FTP client, but basically the FTP protocol allows you to execute synchronous commands. To allow concurrent uploads / downloads, you need to open several FTP connections. This can be done using ftplib and threads.

Answer 2

Another approach to concurrency, in addition, is asynchronous io. For Python, Twisted is the standard toolkit for asynchronous networking .

Take a look at this example FTP client in Twisted.

Answer 3

, - pycurl.

, , , .

: http://pycurl.cvs.sourceforge.net/pycurl/pycurl/examples/

retriever-multi.py file_upload.py , .

Answer 4

You can run the script in several command prompts / shells (just make sure that each file is processed once by all different scripts). I'm not sure if this quick and dirty trick will improve the transfer speed though ..