Take a look at multiprocessor mode?

Question

I have several workflows that are read from the same multiprocessing.queue(). Each workflow only reads content that belongs to itself and should leave the rest of the content intact. Thus, basically the workflow should first check the contents of the queue and then decide whether to place one item.

Is there any way to do this using multiprocessing.queue?

Answer 1

You can always return messages that you do not need ( if order is not a problem )

def get_my_job():

    while True:
        job = q.get()
        if job == 'mine':
            return job
        q.put(job)
        time.sleep(random()/2) #preventing deadlocks...

, , .

queues = { 'queue4worker_type1': Queue(),
           'queue4worker_type2': Queue(),
          }
#each worker can now consume only messages for its wanted types ...