Run pika ioloop in the background or use custom ioloop

Question

I have a feeling that it’s really not that difficult, but so far I have little success.

Say I have a PikaClass class that wraps pika and provides some business methods.

def PikaClass(object): def __init__(self): # connect to the broker self.connection = pika.SelectConnection(<connection parameters>, self.on_connect) # ..other init stuff.. def on_connect(self, connection): # called when the connection has been established # ..open a channel, declare some queues, etc. def start(self): # start the polling loop self.connection.ioloop.start() def foo(self, **kwargs): # do some business logic, eg, send messages to particular queues 

Intuitively, this is what I would like to achieve: the user creates an instance of PikaClass , sets up a loop going in the background, and then interacts with the object, invoking certain business methods

 p = PikaClass() p.start() bar = p.foo(..) 

The problem is that p.start () blocks and prevents the main code from interacting with the object as soon as start () was called. My first thought was to wrap the call in a thread:

 Thread(target=p.start()).start() bar = p.foo(..) 

But it still blocks and you never get to p.foo (..). The docs mention that you should not share the link between threads so that something could cause a problem.

I also tried using AsyncoreConnection instead of SelectConnection and directly called _connect () (instead of using ioloop), but this has no effect (nothing happens).

So, how can I launch ioloop in the background or at least run my own ioloop?

Note. This is Python 2.6 on win64 (xp) with the latest pika 0.9.4

Answer 1

GIL is not a problem here, because ioloop spends almost all of its time in the select (2) system call, during which the GIL is freed, and other Python threads can start and perform other tasks.

The easiest approach is to establish and demolish a queue connection for each request. You might think that it will be too expensive - because it requires re-authentication and (possibly) re-negotiation of SSL with each connection, but it should be the simplest and most reliable and easiest to write, which should be your controlling factor, if you do not you know that tuning and disabling will actually hurt the performance of your application as a whole (which is best measured by testing).

Another approach would be just start() ioloop after you receive the message to send, and ask for a method that receives a response, stop ioloop so that your program gets control again. You can return ioloop earlier:

  connection.ioloop.poller.open = False 

and then remember to bring it back to True before you call start() again to wait for another answer.

Answer 2

You call "p.start" instead of passing it as a parameter. The code should be:

 Thread(target=p.start).start() 

The thread will call p.start when executing Thread.start.

I am not sure if this will solve your problem, but it may help you find a solution.

Answer 3

You may need a second process, I don’t know much about pika, but if it is pure-python, then it will want to save the GIL - remember that you can only execute one thread per process, no matter how due to limitations in the control counter used for wonderful python garbage collection.

If you start a new process before doing / nothing / else in your if __name__ == "__main__": block and turn it off, waiting for events in the loop, you can send a second process instructions to do part of the work, and then wait for in order to send the result back to you. You will probably want to implement an indexing system so that you can start your work in another process and forget about it until you get an answer.

My only advice if you go down this bath is to avoid side effects, or you have to consider the race conditions, dead ends and all the other terrible things that the GIL protects you from. Send your process to the information he needs before starting it, or make sure that you do not modify it before he asks you about this piece of data.

This is one of those few times when Python will hand you a gun with only a polite note saying that you don’t point it at the foot.