All geek questions in one place

Several consumers and manufacturers connected to the message queue, is this possible in AMQP?

I would like to create a farm of processes that can use OCR text. I was thinking about using a single message queue that is read by several OCR processes.

I would like to make sure that:

  • each message in the queue is ultimately processed
  • work is more or less equally distributed.
  • image will be analyzed by only one OCR process
  • The OCR process will not receive multiple messages at once (so any other free OCR process can process the message).

Can i use AMQP?

I plan to use python and rabbitmq

+4
source share
2 answers

Yes it is possible. Server cluster for real-time MMO. I am working on this method. We use ActiveMQ, but I think that all this is possible with RabbitMQ.

All items that you mentioned, you go out of the box, except the last.

  • each message in the queue is ultimately processed - this is one of the main responsibilities of message brokers
  • work is more or less evenly distributed - this is another :)
  • the image will be analyzed by only one OCR process - for this there is a difference between / topic and / queue. Topics are like broadcast signals; queues are tasks. You need a queue / queue in your script

To do the final job at will, consumers send an AMQ-specific argument when subscribing to a queue:

activemq.prefetchSize: 1

This parameter ensures that the consumer does not receive more messages after he has taken one, and until he sends ack to AMQ. I believe something like this exists in RabbitMQ.

+3
source

Yes, as @nailxx points out. The AMQP programming model is slightly different from JMS in that you only have queues that can be shared between employees or used privately by one employee. You can also easily configure RabbitMQ to use PubSub, or what are called themes in JMS. Go to the Getting Started page on the RabbitMQ website to find a ton of useful information about this.

Now, for your use case, in particular, there are already many tools available. One that people use a lot, and it's well supported, is Celery . Here is a post about him that I think will help you get started:

If you have questions, write to us or send a message to the rabbitmq-discuss mailing list.

+5
source

Source: https://habr.com/ru/post/1299724/

More articles:


All Articles