Send socket object for forked process (multiprocessing.Queue)

Question

I am learning to use WebSockets HTML5, and as part of this I am writing a server in Python so that I can know how they work. I created it the other day, which worked very well, but I wanted to expand it to support multiple endpoints, with each endpoint being a different “service” that can handle web descriptor clients.

My implementation currently works with spawning processes and such (I use multiprocessing instead of streaming, since I read that streaming is not really multi-threading in CPython, and what I think I'm using (default setting to Ubuntu 12.04)), but I had problems sending the received client sockets to the service processes.

This is how I send them (this is done in a loop):

try: #get a new client conn, addr = server.accept() print "Client connected from", addr request = conn.recv(4096) response, close, service = self.handshake(request) conn.send(response) if close: print "Invalid request from", addr conn.close() continue client = WebSockets.WebSocketClient(conn, addr) service.clientConnQueue.put(client) 

'server' is a socket that listens for incoming connections. The handshake method takes care of checking their request and determining which service process should be included in the client. "response" is the HTTP response to send, "close" is True if there was an error, and "service" is a class that inherits from multiprocessing. Queue WebSocktes.WebSocketClient is the place where my send and receive implementation is stored, and it basically works like a shell for a socket.

'clientConnQueue' is created as follows:

 class Service(multiprocessing.Process): """Base class for all services.""" def __init__(self, manager): multiprocessing.Process.__init__(self) self.manager = manager self.clientConnQueue = self.manager.Queue() self.shutdownFlag = multiprocessing.Event() 

manager - multiprocessor .Manager ()

The error I get when I try to put the client in clientConnQueue is as follows:

 File "./WebSocketServer.py", line 183, in <module> main() File "./WebSocketServer.py", line 180, in main server.runServer() File "./WebSocketServer.py", line 67, in runServer service.clientConnQueue.put(client) File "<string>", line 2, in put File "/usr/lib/python2.7/multiprocessing/managers.py", line 758, in _callmethod conn.send((self._id, methodname, args, kwds)) TypeError: expected string or Unicode object, NoneType found 

Then I get an interrupt error on the receiving side.

I got the same error when I used multiprocessing. Queue to send the connection, and I thought that changing it in the queue created by the manager would fix the problem. However, it seems to be doing the same implementation.

Obviously, this is not the way you need to send something similar to this running process, and how to properly send non-serialization objects to the process?

Answer 1

Passing a socket to another process is not a trivial task. For example, look at this question: Is it possible to open a socket and transfer it to another process in Linux

In any case, OS processes or threads are not what you really need to implement the websocket server, due to the large memory overhead. Look at smth with non-blocking sockets ... e.g. Tornado http://www.tornadoweb.org/documentation/websocket.html

Answer 2

This has been for 4+ years, although it does require a bit of work.

Flexibility hangs around in multiprocessing.reduction, and details of the example can be seen in this github gist .

Answer 3

You cannot send sockets to forked processes without using deep unix magic. You can read the book "Advanced Programming" in Unix if you really want to see how it is done in C.

For a simple solution to the problem using streaming, and your code will probably just work. Although threading is not ideal in Python, it is very well suited for heavy use of IO.

Alternatively, create a listening socket before forking it, and listen to all subprocesses ( accept call). Os ensures that only one of them gets the connection. This rule is usually writing servers.

Answer 4

If you create a socket before forcing a process, its file descriptor will be inherited by children.