I have a Perl script that retrieves html pages. I tried rewriting it in Python (Just Coz, I'm trying to learn Python), and I found it to be very slow!
Here's a test script in Perl
#!/usr/bin/perl use LWP::Simple; $url = "http://majorgeeks.com/page.php?id="; open(WEB,">>"."perldata.txt"); for ($column = 1 ; $column <= 20 ; $column ++) { $temp = $url.$column; print "val = $temp\n\n"; $response=get($temp)or die("[-] Failed!!\n"); print WEB "$response\n\n"; }
And here is the equivalent code in Python
import urllib2 url = "http://majorgeeks.com/page.php?id=" f = open("pydata.txt", 'w') for i in range(20): tempurl = url + str(i+1) print "Val : " + tempurl + "\n\n"
The difference I found is huge! The perl script completed in about 30 seconds. While the Python script took 7 min. (420 seconds)!
I use Ubuntu 11.10, 64bit, Core i7, tested it on 12MBPS connection. I tried this several times, and each time I get the same difference.
Am I doing something wrong here? Or do I need to do something? Or is the difference justified? (I hope not)
Many thanks for your help.
Update 3: I just got home and downloaded my laptop, ran the code again and finished it in 11 seconds !!!: / Is it because I "rebooted" my computer? Here is the Profiler output
Note. Perl still takes 31 seconds to do the same!: /
Update 2: As suggested by @Makoto Here are the profiler data I made. And it is very slow! I know some kind of python configuration is related to this, but I don't know what. For one simple request, it should not take 20 seconds.
Update: Fixed url for tempurl. Commented on urllib2.Request as suggested here. Not a big difference.