Threading issues with xmlrpclib.ServerProxy under python 2.7

Question

I have an application that instantiates xmlrpclib.ServerProxy once and then passes it to multiple threads (web application requests) that make XML / RPC calls at the same time. This works well with python 2.6. With python 2.7, we get a lot of errors ( ResponseNotReady , CannotSendRequest ) as soon as we are in a multi-threaded environment.

 # This code works well in python 2.6, and breaks in python 2.7. import xmlrpclib import thread proxy = xmlrpclib.ServerProxy("http://localhost:5000/") def fetch_users(): print proxy.getUsers() for _ in range(10): thread.start_new_thread(fetch_users, ()) while(1): pass 

What is the problem, is there a thread-safe way to reuse a ServerProxy object?

Answer 1

We found the cause of the problem: in python 2.6, a TCP connection is created each time the XML / RPC method is called. Python 2.7, on the other hand, opens one TCP connection for each ServerProxy object and keeps it open (with servers that support keep-alive ). In addition, the class is not thread safe, so simultaneous requests can interfere with each other.

Version 2.6 was apparently thread-safe implicitly, since TCP connections were not reused, and all data related to the connection was apparently stored in variables that were not connected to the common stack.

Thus, possible solutions:

• Create a ServerProxy object (and implicitly open a TCP connection) for each thread
• Block access to one shared ServerProxy object
• Request Queue Implementation

Answer 2

Most code is not thread safe. However, I do not know why the code worked in version 2.6, but would raise errors in 2.7.

Here is another question:

• uses a higher level threading module

• A proxy server is a stream, not a global one. This is safer, so threads do not mix each other, overwriting a common global object.

• explicitly explicitly join() ed at the end to make sure they are all completed

A source

 import xmlrpclib import threading def fetch_users(): proxy = xmlrpclib.ServerProxy("http://localhost:5000/") print proxy.getUsers() for _ in range(10): threading.Thread(target=fetch_users, args=()).start() # wait for all threads to exit for th in threading.enumerate(): th.join()