How can I force a socket opened with python closure?

Question

I am currently using this lib for stress testing the kafka server I created: https://github.com/dsully/pykafka

import kafka import time def test_kafka_server(n=1): for i in range(0,n): producer = kafka.producer.Producer('test',host='10.137.8.192') message = kafka.message.Message(str(time.time())) producer.send(message) producer.disconnect() def main(): test_kafka_server(100000) if __name__ == '__main__': main() 

What is just ending is that I am overloading my local machine.

I get error 10055 , which, according to Google, means that "Windows ended the TCP / IP socket buffers because there are too many connections at once." According to netstat, the manufacturer .disconnect () does not close the socket, but transfers it to the TIME_WAIT state.

The ipython debugger points to this line:

 C:\Python27\lib\socket.pyc in meth(name, self, *args) 222 proto = property(lambda self: self._sock.proto, doc="the socket protocol") 223 --> 224 def meth(name,self,*args): 225 return getattr(self._sock,name)(*args) 226 

like a criminal, but it seems to interact with things at a lower level than I'm comfortable with.

I searched and found that this Python socket does not close the connection properly , which is recommended:

 setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 

So, I rebuilt libkpka lib using this option in the io.py file:

  def connect(self): """ Connect to the Kafka server. """ global socket self.socket = socket.socket() self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.socket.connect((self.host, self.port)) 

and I still get the same error.

I do not put the setsockopt line in the right place? Is there anything else I could try?

Answer 1

What you are describing is the normal TCP behavior at the socket level. When a user-level program closes a socket, the kernel does not immediately release the socket. It enters TIME_WAIT state:

TIME-WAIT (either the server or the client) represents an expectation sufficient to pass to make sure that the remote TCP has received confirmation of its connection termination request. [According to RFC 793, a compound can remain in TIME-WAIT for a maximum of four minutes, known as MSL (maximum segment lifetime).

So, the connector is closed. The .SO_REUSEADDR socket is intended for listeners (servers), does not affect client connections. Well, really used when binding a socket.