Handling HTTP Authentication When Accessing Remote URLs Through Pandas

Question

Pandas has a very convenient ability to read csv and other formats from URLs. However, when the data is protected by simple HTTP authentication, Pandas cannot ask the user for authentication data (userid, password). What is the best way to fix this restriction?

what am i doing now:

response = requests.get('http://my.data.url/metrics/crawler/counts', auth=HTTPBasicAuth('userid', 'password'), stream=True) pd.read_csv(response.raw) 

is there a better way?

Answer 1

It is not possible to test, but it looks like basic access authentication can be encoded in the url. From wikipedia :

URL encoding

The client can avoid the login prompt when accessing basic access authentication by adding the username: password @ to the host name in the URL. For example, the following will access the index.html page on www.example.com using the secure HTTPS protocol and provide the Aladdin username and password for the OpenSesame credentials using basic authorization: https: // Aladdin: [e-mail] /index.html

Hope this helps!