Configure cluster-based message rate

Question

I am writing a simple homogeneous cluster application using Akka 2.2.3 and Scala; a particle filtering algorithm in which each node shares data with other members of the cluster at random times. This is currently a research application, not a business critical system.

Currently, each node sends a fixed-size message to a randomly selected node every second. This works, but I have performance issues when scaling (e.g. cloud versus on-premises)

• Nodes can overload sent data
• Nodes can be overloaded with incoming messages from other members of the cluster.
• The network may become a bottleneck

I would like to run an application with cluster sizes on different networks and achieve good performance without manual tuning / monitoring. What simple approaches can be used to adjust the size and frequency of messages to mitigate the above problems?

Answer 1

You can try using Adaptive Load Balancing routers that support the Akka cluster.

Or you can try to create your own mechanism to reduce node congestion. For example, it may contain a custom mailbox for your receiver, which can periodically add a message containing the current queue of the mailbox queue to the head of the mailbox queue. Then your receiver can transmit such a message to all nodes in the cluster. In this case, you will be able to maintain statistics on the use of mailboxes within the cluster (just remember to include TTL in these statistics and not process the statistical message received as an indicator of an empty mailbox).

If your application can tolerate partial data loss, you can simply drop additional messages directly from Receiver mailboxes (you will also need to implement a custom mailbox).

Answer 2

If this is possible with your algorithm, another approach would be to rotate the case and the nodes to ask other random nodes for the data at random times. Working tension is usually easier to control than to press.