How to specify authenticated proxy for python http connection?

Question

What is the best way to specify a proxy with username and password for http connection in python?

Answer 1

Use this:

import requests proxies = {"http":"http://username:password@proxy_ip:proxy_port"} r = requests.get("http://www.example.com/", proxies=proxies) print(r.content) 

I think this is a lot easier than using urllib . I do not understand why people like to use urllib so much.

Answer 2

This works for me:

 import urllib2 proxy = urllib2.ProxyHandler({'http': 'http:// username:password@proxyurl:proxyport'}) auth = urllib2.HTTPBasicAuthHandler() opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler) urllib2.install_opener(opener) conn = urllib2.urlopen('http://python.org') return_str = conn.read() 

Answer 3

Set up the var environment with the name http_proxy as follows: http: // username: password @proxy_url: port

Answer 4

The best way for a proxy server to require authentication is to use urllib2 to create a custom url opener, and then use this to create all the requests you want to go through the proxy. Please note in particular that you probably do not want to embed a proxy password in the python url or source code (unless it's just a quick hack).

 import urllib2 def get_proxy_opener(proxyurl, proxyuser, proxypass, proxyscheme="http"): password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() password_mgr.add_password(None, proxyurl, proxyuser, proxypass) proxy_handler = urllib2.ProxyHandler({proxyscheme: proxyurl}) proxy_auth_handler = urllib2.ProxyBasicAuthHandler(password_mgr) return urllib2.build_opener(proxy_handler, proxy_auth_handler) if __name__ == "__main__": import sys if len(sys.argv) > 4: url_opener = get_proxy_opener(*sys.argv[1:4]) for url in sys.argv[4:]: print url_opener.open(url).headers else: print "Usage:", sys.argv[0], "proxy user pass fetchurls..." 

In a more complex program, you can separate these components as needed (for example, using only one password manager for the life of the application). The python documentation has more examples of how to do complex things with urllib2 , which can also be useful.

Answer 5

Or, if you want to install it so that it is always used with urllib2.urlopen (so you do not need to store a link to the opener):

 import urllib2 url = 'www.proxyurl.com' username = 'user' password = 'pass' password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() # None, with the "WithDefaultRealm" password manager means # that the user/pass will be used for any realm (where # there isn't a more specific match). password_mgr.add_password(None, url, username, password) auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr) opener = urllib2.build_opener(auth_handler) urllib2.install_opener(opener) print urllib2.urlopen("http://www.example.com/folder/page.html").read() 

Answer 6

The urllib method is used here

 import urllib.request # set up authentication info authinfo = urllib.request.HTTPBasicAuthHandler() proxy_support = urllib.request.ProxyHandler({"http" : "http://ahad-haam:3128"}) # build a new opener that adds authentication and caching FTP handlers opener = urllib.request.build_opener(proxy_support, authinfo, urllib.request.CacheFTPHandler) # install it urllib.request.install_opener(opener) f = urllib.request.urlopen('http://www.python.org/') """