Help with cURL in Python

Question

I need to send a request to the server. There is this example in the API documentation on the website that uses cURL in PHP:

$ch = curl_init(); curl_setopt($ch, CURLOPT_URL, 'http://api.website.com'); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, "request=$wrapper"); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); ; $data = curl_exec($ch); curl_close($ch); 

But my application is executed using Python, so I tried to write something like this, but this code does not work:

 req = urllib2.Request(url, formatted) response = urllib2.urlopen(req) html = response.read() print html+"\n\n" 

Can you help me write a working conversion of a PHP cURL program to Python?

Thanks!

Answer 1

This is pretty confusing, but ... the only problem with my code that uses urllib and urllib2 is ... that this code does GET, not POST !!!

Here is my scan using Wireshark:

1- using urllib and urllib2

 Hypertext Transfer Protocol GET / HTTP/1.1\r\n [Expert Info (Chat/Sequence): GET / HTTP/1.1\r\n] [Message: GET / HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: GET Request URI: / Request Version: HTTP/1.1 Accept-Encoding: identity\r\n Host: api.apptrackr.org\r\n Connection: close\r\n User-Agent: Python-urllib/2.6\r\n \r\n 

2- using PyCurl

 Hypertext Transfer Protocol POST / HTTP/1.1\r\n [Expert Info (Chat/Sequence): POST / HTTP/1.1\r\n] [Message: POST / HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: POST Request URI: / Request Version: HTTP/1.1 User-Agent: PycURL/7.19.5\r\n Host: api.website.com\r\n Accept: */*\r\n Content-Length: 365\r\n [Content length: 365] Content-Type: application/x-www-form-urlencoded\r\n \r\n Line-based text data: application/x-www-form-urlencoded [truncated] request=%7B%22enc_key%22%3A%22o37vOsNetKgprRE0VsBYefYViP4%2ByB3pjxfkfCYtpgiQ%2ByxONgkhhsxtqAwaXwCrrgx%2BPDuDtMRZNI1ez//4Zw%3D%3D%22%2C%22format%22%3A%22RSA_RC4_Sealed%22%2C%22profile%22%3A%22Ldn%22%2C%22request%22%3A%22bQ%2BHm/ 

so the code works, but it’s not so for me, because I need POST, but I prefer to use NOT PyCurl. Any idea?

Many thanks!

Answer 2

curl also for Python: http://pycurl.sourceforge.net/

The example can be translated into Python and pycurl as follows:

 import pycurl c = pycurl.Curl() c.setopt(pycurl.URL, "http://api.website.com") c.setopt(pycurl.POST, 1) c.setopt(pycurl.POSTFIELDS, "request=%s" % wrapper) import StringIO b = StringIO.StringIO() c.setopt(pycurl.WRITEFUNCTION, b.write) c.perform() c.close() data = b.getvalue() 

Your Python code using urllib2 looks fine, it should work. There is probably a mistake in something else that you were not talking about; could you be more specific?

Answer 3

Taking your code, this should really work.

  import urllib import urllib2 url = 'http://api.website.com/' values = {'some_key':'some_value'} data = urllib.urlencode(values) req = urllib2.Request(url, data) response = urllib2.urlopen(req) page = response.read() print page + '\n\n' 

What error do you get?

Answer 4

Think about how to use the packet sniffer to find out if cURL sends user agent information. If so, and the service expects this information, then use the add_header () method on Request (from the urllib2 documentation at the bottom of the page):

 import urllib2 req = urllib2.Request('http://api.website.com/') # Your parameter encoding here req.add_header('User-agent', 'Mozilla/5.0') r = urllib2.urlopen(req) # Process the response