Design and Scale the RabbitMQ Queue

Question

In my application, I have a queue that could potentially become very large. What if I find that there is no more space on my machine? How can I split a queue on multiple machines? Perhaps the philosophy of RabbitMQ is different, and I should create multiple queues instead of one large queue.?

Best, Flavio

Answer 1

RabbimMQ provides clustering and high availability features right out of the box, you just need to customize them to your needs.

In fact, AMQP can store messages of any size, but in most cases messages may be up to 32 KB in 99%. You can calculate the use of evaluation resources (min / max / avg) and make further decisions about the cluster or non-cluster.

Answer 2

As you can read on the RabbitMQ mailing list http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-scalability-design-question-td28323.html , the solution I came up with is to implement a competing consumer template (many consumers in queue) that when a special message is detected (with the stop flag turned on) it sends a STOP message to exchange the topic.

This stop message is received by the "main" consumer for this queue, which starts polling Zookeeper (via the curator) until all zarNode ceratain child nodes are deleted (using Zookpeer as the queue user registry in this case), when all consumers have completed its stop phase, the "main" consumer performs some task and re-activates the initial consumers of the queue by sending a RESTART message to exchange threads (where each consumer listens with a dedicated queue).

I hope this can help (or "inspire") someone else.