Using tor as a SOCKS5 proxy with python urllib2 or mechanize

Question

My goal is to use python mechanics with SOCKS proxies.

I do not use the GUI with the next version of Ubuntu: Description: Ubuntu 12.04.1 LTS Issue: 12.04 Codename: accurate

Tor is installed and listens on port 9050 according to nmap scan:

Starting Nmap 5.21 ( http://nmap.org ) at 2013-01-22 00:50 UTC Nmap scan report for localhost (127.0.0.1) Host is up (0.000011s latency). Not shown: 996 closed ports PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 3306/tcp open mysql 9050/tcp open tor-socks 

I also found it reasonable to see if I can use telnet for port 9050, which I can:

  telnet 127.0.0.1 9050 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. quit Connection closed by foreign host. 

I had high hopes for a suggestion in this post to make working with urllib2: How to use SOCKS 4/5 proxy with urllib2?

So, I tried the following script in python:

  import socks import socket socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050) socket.socket = socks.socksocket import urllib2 print urllib2.urlopen('http://icanhazip.com').read() 

The script just hangs without an answer.

I thought that since mechanization seems to be related to urllib2, the following script might work:

  import socks import socket import mechanize from mechanize import Browser socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050) socket.socket = socks.socksocket br = Browser() print br.open('http://icanhazip.com').read() 

I get the same result as above with urllib2 script.

I am very new to python and the web, so I need a second opinion on how to get python urllib2 to use tor as SOCKS on a server without a Ubuntu GUI.

I ran this script and got the expected response. I did not use tor proxy:

  In [1]: import urllib2 In [2]: print urllib2.urlopen('http://icanhazip.com').read() xxxx:xxxx:xxxx:512:13b2:ccd5:ff04:c5f4 

Thank.

I found something that works! I have no idea why this works, but it does. I found it here: Python urllib over TOR?

  import socks import socket def create_connection(address, timeout=None, source_address=None): sock = socks.socksocket() sock.connect(address) return sock socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050) # patch the socket module socket.socket = socks.socksocket socket.create_connection = create_connection import urllib2 print urllib2.urlopen('http://icanhazip.com').read() import mechanize from mechanize import Browser br = Browser() print br.open('http://icanhazip.com').read() 

Answer 1

See the end of the question.

 import socks import socket def create_connection(address, timeout=None, source_address=None): sock = socks.socksocket() sock.connect(address) return sock socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050) # patch the socket module socket.socket = socks.socksocket socket.create_connection = create_connection import urllib2 print urllib2.urlopen('http://icanhazip.com').read() import mechanize from mechanize import Browser br = Browser() print br.open('http://icanhazip.com').read() 

Answer 2

The above solution did not work for me. I'm on Ubuntu 14.04. Whenever I try to run the above script, it keeps throwing the following error.

 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 404, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 422, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open return self.do_open(httplib.HTTPConnection, req) File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open raise URLError(err) urllib2.URLError: <urlopen error ((1, 'general SOCKS server failure'),)> 

Checked if tor works with nmap.

 Nmap scan report for localhost (127.0.0.1) Host is up (0.00026s latency). Not shown: 993 closed ports PORT STATE SERVICE 22/tcp open ssh 139/tcp open netbios-ssn 445/tcp open microsoft-ds 631/tcp open ipp 902/tcp open iss-realsecure 3306/tcp open mysql 9050/tcp open tor-socks 

Installing Vidalia solved this problem. Apparently, the socks proxy did not allow the connection to pass through it. Hope this helps someone who is facing the same problem.