LWP Result :: Simple get truncated

Question

I use perl to execute a get request on url and the results seem to be truncated.

If I run

curl myurl | wc -l 

the result is 1823, and if I create the following foo.pl file:

 #!/usr/bin/perl my $url = 'myurl'; use LWP::Simple; my $content = get $url; die "Couldn't get $url" unless defined $content; print $content; 

and run

 ./foo.pl | wc -l 

the result varies from about 1300 to sometimes 1823. Manually checking the output shows that the output is interrupted in the middle of the line when using perl.

What could be the reason for this?

Answer 1

What happens if you turn off buffering? I also agree with Karsten S. in checking http headers for erroneous codes. Finally, I also tried storing the contents in an array to see what happens.

To disable buffering, you can simply put $|++ at the top of your script after your use statements. Again, a shot in the dark.

To learn http headers, you can use CGI . Here is a small site with a good example of how to get the headers from the request:

http://www.velocityreviews.com/forums/t24118-re-lwp-simple-header-information-problems.html

Finally, try using the @contents array to store content from the web server instead of the $contents scalar. I used to have a time when something is being transferred from a remote server, which Perl incorrectly interprets as a list. I am not sure if LWP::Simple is responsible for these times, but cannot stop trying. You can only get one piece of data, and the rest will either be overwritten or ignored altogether. Placing data in an array can help determine if this is happening.