How to make parallel HTTP requests in Perl and get them back in order?

Question

Using Perl, I am looking for an easy way to execute multiple HTTP requests in parallel, where I get the responses back in the same order that I sent them after they completed, for example:

my ($google, $perl) = foobar(GET => 'http://www.google.com/', GET => 'http://www.perl.org/'); 

Is there a module I should look at?

I know I can do financial statements by hand, but I feel spoiled after being able to do this using the jQuery when method , and I would like to start using Perl was a simple decision.

Thank you for your help.

Answer 1

 use threads; use LWP::UserAgent qw( ); my $ua = LWP::UserAgent->new(); my @threads; for my $url ('http://www.google.com/', 'http://www.perl.org/') { push @threads, async { $ua->get($url) }; } for my $thread (@threads) { my $response = $thread->join; ... } 

The best part is that the parent does not wait for all requests to complete. Once the correct request is complete, the parent will be unlocked to process it.

---

If you used Parallel :: ForkManager or something else where you cannot wait for a specific child, you can use the following code to order the results:

 for my $id (0..$#urls) { create_task($id, $urls[$id]); } my %responses; for my $id (0..$#urls) { if (!exists($responses{$id})) { my ($id, $response) = wait_for_a_child_to_complete(); $responses{$id} = $response; redo; } my $response = delete($responses{$id}); ... } 

Answer 2

I am a fan of Mojo ! From the Mojo :: UserAgent document :

 use Mojo; use Mojo::UserAgent; # Parallel requests my $ua = Mojo::UserAgent->new; $ua->max_redirects(5); my $delay = Mojo::IOLoop->delay; for my $url ('http://www.google.com/', 'http://www.perl.org/') { $delay->begin; $ua->get($url => sub { my ($ua, $tx) = @_; $delay->end($tx->res->dom); }); } my @responses = $delay->wait; print join "\n", @responses 

Enjoy it!

EDIT

Btw. you do not need to process the answers at the end, you can do this between them:

 # ... $ua->get($url => sub { my ($ua, $tx) = @_; $delay->end(1); # process $tx->res here }); # ... $delay->wait;