Simple python proxy - almost working

Question

I am working on a slightly larger project and I need to make a localhost proxy in python.

The way I wrote is that there is a TCP server on port 8080 on the local host (using socket and SOCK_STREAM). It receives a request from the local host using slicing, string.find () and gethostbyname () finds the destination IP address, so it opens another TCP socket, sends the request and returns a response. After that, it returns the response back to the localhost proxy, which, in turn, returns it to the browser.

This is code with enough debugging messages and a debug file to collect browser requests and received responses (also note that this is just a prototype, therefore, a limited while loop instead of a while loop):

import socket local = socket.socket(socket.AF_INET, socket.SOCK_STREAM) f = open('test.txt', 'a') local.bind(('localhost', 8080)) local.listen(5) for i in xrange(20): print '=====%d=====\n' % i out = socket.socket(socket.AF_INET, socket.SOCK_STREAM) data, addr = local.accept() print 'Connection accepted' buffer = data.recv(4096) print 'data recieved' f.write('=============================================================\n') f.write(buffer) end = buffer.find('\n') print buffer #print buffer[:end] host = buffer[:end].split()[1] end = host[7:].find('/') print host[7:(end+7)] host_ip = socket.gethostbyname(host[7:(end+7)]) #print 'remote host: ' + host + ' IP: ' + host_ip print 'sending buffer to remote host' out.connect((host_ip, 80)) out.sendall(buffer) print 'recieving data from remote host' reply = out.recv(4096) out.close() print 'data recieved from remote host' f.write('+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n') f.write(reply) f.write('\n\n\n') print 'sending data back to local host' data.sendall(reply) print 'data sent' local.close() out.close() f.close() 

Now my problem is that it works fine for the first few requests, it receives html and several images, but at some point it always stops at the “received” point and exits because it does not receive data, i.e. The buffer is empty. The browser still shows the page load elements, but when it stops and I look at the text log file, I see that the buffer is empty, which means that the browser did not transmit anything to the proxy server?

I assume the problem is how the browser sends requests, and my script is not responding properly to this behavior.

I know that I can use the Twist framework, but I want to learn how to write this material myself. I read about SocketServer and I could use this, but I don’t know if it will solve the problem, because, to be honest, I really don’t understand what causes the problem here. Is my script too slow for the browser? Do the servers send multiple responses and should my receive socket listen on more packets? Is my buffer size (4096) too small?

I would really like to push you in the right direction.

Thanks!

Answer 1

Well, I managed to answer my question. What I suspected earlier was partly true - the browser was waiting for something and responding.

I ran a wired shark, did some experimentation, and I noticed that my proxy makes a lot of ugly TCP RSTs appearing in wirehark. I also noticed that in a regular connection, many server responses are split into several different packets.

Basically, my program did not receive responses from the server because out.recv received only one part of the response. The obvious answer was to create a loop and listen to all the answers. I found the perfect solution at http://www.binarytides.com/receive-full-data-with-the-recv-socket-function-in-python/ .

I quickly transcoded my program and it works like a charm. Now I can move forward with my entire project.

I hope this helps someone else in the future with a similar problem.