Gevent threads do not end although all queue elements are exhausted

Question

I am trying to create a simple producer-consumer system in Gevent, but my script does not exit:

import gevent from gevent.queue import * import time import random q = Queue() workers = [] def do_work(wid, value): """ Actual blocking function """ gevent.sleep(random.randint(0,2)) print 'Task', value, 'done', wid return def worker(wid): """ Consumer """ while True: item = q.get() do_work(wid, item) def producer(): """ Producer """ for i in range(4): workers.append(gevent.spawn(worker, random.randint(1, 100000))) for item in range(1, 9): q.put(item) producer() gevent.joinall(workers) 

I could not find good examples / tutorials on using Gevent, so what I pasted above is what I exhausted from the Internet.

Several workers are activated, items enter the queue, but even when everything in the queue ends, the main program does not exit. I have to press CTRL ^ C

What am I doing wrong?

Thanks.

On the side of the note: if there is something my script can improve, let me know. Simple things like checking when the queue is empty, etc.

Answer 1

I think you should use JoinableQueue , as in the example from the documentation.

 import gevent from gevent.queue import * import time import random q = JoinableQueue() workers = [] def do_work(wid, value): gevent.sleep(random.randint(0,2)) print 'Task', value, 'done', wid def worker(wid): while True: item = q.get() try: do_work(wid, item) finally: q.task_done() def producer(): for i in range(4): workers.append(gevent.spawn(worker, random.randint(1, 100000))) for item in range(1, 9): q.put(item) producer() q.join() 

Answer 2

In your desktop, you activate a loop that will last forever.

As a side note, the more elegant "endless loop" imho can be written simply:

 for work_unit in q: # Do work, etc 

gevent.joinall () awaits completion by employees; but they never do this, so your program will wait forever. This is what makes him stay out.

If you no longer care about the workers, you can simply kill them:

 gevent.killall(workers) 

An alternative is to queue a "special" item. When an employee receives this item, he recognizes it as different from normal work and stops working.

 for worker in workers: q.put("TimeToDie") for work_unit in q: if work_unint == "TimeToDie": break do_work() 

Or you can even use gevent Event to create such a template.