How can I resume loading in Perl?

Question

I have a project that depends on some other binaries that will be downloaded from the Internet during installation. For this I do:

if ( file-present-in-src/) # skip that file else # use wget to download the file 

The problem with this approach is that when I interrupt the download in the middle and the script is called the next time, the partially downloaded file is also skipped (which is undesirable), I also want wget to resume downloading the partially downloaded file.

How do I do this: Possible solutions that I could think of:

• Let the file be loaded into some say download_tmp file. Move to source file if successful.
• Process SIG {'INT'} to write the correct cleanup code.

But none of them can help resume downloading a partial file,

Any ideas?

Answer 1

Fist, I don’t understand that this is related to Perl, since you are using wget to load dowloading ... You can use libwww-perl (perldoc LWP) and have more control over the loading process.

Then I will force your idea of ​​loading into the "tmp" file name and move the file successfully.

However, I think you need to go further and check the integrity of the files. Executing an MD5 or SHA hash is very simple and matches the loaded what you expect. You may have a short file on the server containing the checksum (filename.md5). Define success only when you have a match.

Please note that the capture of all signals and, as a rule, an attempt to make the process unsatisfactory, and then expect that it will work, at some point, will work. There may be a network timeout, a power failure, a power failure, a configuration problem on the server ... you must assume that the download may fail because they will, and the code so that your process can recover.

Finally, you do not tell us which binary files you download and what you do with them. Since you are using wget, I am going to assume that you are working on Unix; you should consider using RPM + Yum or the like, they will handle all this for you. RPM is easy to write, really.

Answer 2

use your first approach.

• download to "FileName" .tmp
• move "FileName" .tmp to "FileName"
  move! do not copy
• clean all .tmp files once a day (paranoia rulez)

Answer 3

You can simply use the wget -N and -c options and remove all the "if there is a file" logic.