Curl works, but not Python requests

Question

I am trying to get a JSON response from http://erdos.sdslabs.co/users/shagun.json . Using the browser library / Python Requests results in an authentication error, but curl seems to work fine.

curl http://erdos.sdslabs.co/users/shagun.json 

returns a JSON response.

Why does a curl request fire when a regular browser or request-based request fails?

Answer 1

Using telnet to verify:

 $ telnet erdos.sdslabs.co 80 Trying 62.141.37.215... Connected to erdos.sdslabs.co. Escape character is '^]'. GET http://erdos.sdslabs.co/users/shagun.json HTTP/1.0 HTTP/1.1 302 Found Date: Sat, 26 Jul 2014 11:18:58 GMT Server: Apache Set-Cookie: PHPSESSID=juvg7vrg3vs4t00om3a95m4sc7; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Location: /login Access-Control-Allow-Origin: http://erdos.sdslabs.co X-Powered-By: PleskLin Content-Length: 1449 Connection: close Content-Type: application/json {"email":" [e-mail] ","username":"shagun","name":"Shagun [...] 

We see that the web server responds with 302 - redirecting to Location / login. Requests and web browsers obey this and reach a login prompt. However, we see that the web server is also responsible for the json that you are after, and the curl (and telnet) are simple enough to just accept this data.

The best practice would be to fix the web server so that it either does not require logging in or does not produce password-protected data at the same time as users are asked to enter.

If you cannot change the web server, you can tell the request module to ignore the redirects:

 import requests result = requests.get('http://erdos.sdslabs.co/users/shagun.json', allow_redirects=False) print result.content 

Answer 2

If you have a proxy server configured in your environment, define it in your session / request.

For example, with a session:

  my_proxies = { 'http': 'http://myproxy:8080', 'https': 'https://myproxy:8080' } session = requests.Session() request = requests.Request('POST', 'http://my.domain.com', data=params_template, headers=req_headers, proxies=my_proxies) prepped = session.prepare_request(request) response = session.send(prepped) 

see documentation:
request http://docs.python-requests.org/en/master/user/quickstart/
session http://docs.python-requests.org/en/master/user/advanced/