Download only new / changed files using perl (or wget)

Question

I have a Perl script that downloads a large number of files from a remote server. I would like to avoid hacking the server, so I would like to avoid downloading the file if it has not been modified since the last check. Is there a good way to do this, either in Perl or with a shell script?

Can I force the server to send HTTP 304 and not HTTP 200 for unmodified files?

Answer 1

Yes, use LWP::UserAgentand pay particular attention to the method mirror. It is also available in procedural LWP::Simpleas a function mirror.

LWP POD:

, $url, $filename. , "If-Modified-Since", . , . , . .

.

HTTP 304 - , , If-Modified-Since, . LWP mirror - .

Answer 2

, , - . ; , .

#!/usr/bin/perl -w

require HTTP::Date;
require LWP::UserAgent;
require Date::Parse;

my $lastChecked = '2009-01-01';
my $ua = LWP::UserAgent->new;
$ua->default_header('If-Modified-Since' => HTTP::Date::time2str(Date::Parse::str2time($lastChecked)));

my $response = $ua->get('http://example.com/');

if ($response->code == 304) {
    print "No changes.\n";
} elsif ($response->is_success) {
    print $response->decoded_content;
} else {
    print "Response was error " . $response->code . ": '" . $response->status_line . "'\n";
}