All geek questions in one place

Resume FTP upload after timeout

I upload files from a flaky FTP server, which is often missing during file transfer, and I was wondering if there is a way to reconnect and resume the download. I am using python ftplib. Here is the code I'm using:

#! /usr/bin/python import ftplib import os import socket import sys #--------------------------------# # Define parameters for ftp site # #--------------------------------# site = 'a.really.unstable.server' user = 'anonymous' password = ' someperson@somewhere.edu ' root_ftp_dir = '/directory1/' root_local_dir = '/directory2/' #--------------------------------------------------------------- # Tuple of order numbers to download. Each web request generates # an order numbers #--------------------------------------------------------------- order_num = ('1','2','3','4') #----------------------------------------------------------------# # Loop through each order. Connect to server on each loop. There # # might be a time out for the connection therefore reconnect for # # every new ordernumber # #----------------------------------------------------------------# # First change local directory os.chdir(root_local_dir) # Begin loop through for order in order_num: print 'Begin Proccessing order number %s' %order # Connect to FTP site try: ftp = ftplib.FTP( host=site, timeout=1200 ) except (socket.error, socket.gaierror), e: print 'ERROR: Unable to reach "%s"' %site sys.exit() # Login try: ftp.login(user,password) except ftplib.error_perm: print 'ERROR: Unable to login' ftp.quit() sys.exit() # Change remote directory to location of order try: ftp.cwd(root_ftp_dir+order) except ftplib.error_perm: print 'Unable to CD to "%s"' %(root_ftp_dir+order) sys.exit() # Get a list of files try: filelist = ftp.nlst() except ftplib.error_perm: print 'Unable to get file list from "%s"' %order sys.exit() #---------------------------------# # Loop through files and download # #---------------------------------# for each_file in filelist: file_local = open(each_file,'wb') try: ftp.retrbinary('RETR %s' %each_file, file_local.write) file_local.close() except ftplib.error_perm: print 'ERROR: cannot read file "%s"' %each_file os.unlink(each_file) ftp.quit() print 'Finished Proccessing order number %s' %order sys.exit()

The error I get is: socket.error: [Errno 110] Connection timeout

Any help is greatly appreciated.

+6
source share
2 answers

Resuming FTP downloads using only standard tools (see RFC959 ) requires the use of block transfer mode (section 3.4.2), which can be set using the MODE B command. Although this feature is technically necessary to meet specifications, I am not sure that all FTP server software implements it.

In block transfer mode, in contrast to stream transfer mode, the server sends the file to pieces, each of which has a marker. This token can be re-sent to the server to restart the failed transfer (section 3.5).

The specification states:

[...] a restart procedure is provided to protect users from system failures (including failures of the host, FTP process or core network).

However, AFAIK, the specification does not define the required lifetime for markers. He says only the following:

Token information is meaningful only to the sender, but should consist of printed characters in a standard or coordinated control connection language (ASCII or EBCDIC). The token may represent a bit count, a record counter, or any other information by which the system can identify a data checkpoint. The data receiver, if it implements the restart procedure, then marks the corresponding position of this marker in the receiving system and returns this information to the user.

You must be sure that servers that implement this feature will provide tokens that are valid between FTP sessions, but your mileage may vary.

+3
source

To do this, you will need to save the interrupted download, then find out which parts of the file you are missing, download these parts, and then connect them together. I'm not sure how to do this, but there is a download manager for Firefox and Chrome called DownThemAll that does this. Although the code is not written in python (I think it's JavaScript), you can look at the code and see how it does it.

DownThemll - http://www.downthemall.net/

0
source

Source: https://habr.com/ru/post/894219/

More articles:


All Articles