Best practice for upgrading a node in a load balanced cluster in a rabbitMQ cluster?

Question

Summary: what is best for updating a node in a balancing cluster?

We use the RabbitMQ cluster behind the ha proxy load balancer to support simple clustering for our clients as suggested in the RabbitMQ docs

Although the docs suggest this, they do not describe the best way to remove a node from the cluster for updates and bring it back.

Here is the process that I think we should use:

• remove the node from the cluster by running rabbitmqctl stop_app in the node itself and wait for it to complete
• put node in maint mode in haproxy
• execute maint work
• attach the node to the cluster, confirm that it joins and synchronizes.
• remove node from maint mode in haproxy

but I suggested that we first remove it from ha-proxy, basically changing steps 1 and 2 above

Here is the process suggested by another team member:

• put node in maint mode in haproxy
• remove the node from the cluster by running rabbitmqctl stop_app in the node itself and wait for it to complete
• execute maint work
• attach the node to the cluster, confirm that it joins and synchronizes.
• remove node from maint mode in haproxy

What is the best way to do this?

Answer 1

For me, the obvious way would be to tell you about haproxy that you want to stop sending requests to the server, and then stop the server itself, and not vice versa.

I am wondering why you want to stop the server first and then put it in maint? If you do this like this, some requests will be sent to your node until it is known that it is gone. I believe that you can configure haproxy to resubmit these missed calls; therefore, at best, you have several queries that are slightly slower, at worst, you have missed requests.

There is no specific flaw that I can see with setting it up in maint mode first so that I do not consider another option personally.