Tuning message frequency based on cluster load

Question

I am writing a simple homogenous cluster application using Akka 2.2.3 and Scala; a particle filtering algorithm in which each node shares data with other cluster members at random points in time. It's currently a research application, not a business critical system.

At present, every node sends a fixed size message to a randomly selected node each second. This works, but I have concerns regarding performance when scaling (e.g. cloud versus local)

• Nodes may get overloaded sending data
• Nodes may get overloaded with incoming messages from other cluster members
• The network may become the bottleneck

I'd like to run the application with sized clusters on different networks and and achieve good performance without manual tuning/monitoring. What simple approaches could I take to tuning message size and frequency to mitigate the above concerns?

Answer 1

You can try to use Adaptive Load Balancing of Akka cluster-aware routers.

Or you can try to create your own mechanism for reduce nodes overload. For example it may contain a custom mailbox for your Receiver actor which may periodically add a message containing the current mailbox queue length to the head of the mailbox queue. Then your Receiver actor may relay such a message to all cluster nodes. In this case you will be able to maintain cluster-wide statistics of Receiver's mailboxes usage (just remember to include TTL in that statistics and treat no statistics message received as empty mailbox indicator).

If your application can tolerate partial data losses then you can simply drop extra messages directly from the Receiver's mailboxes (as well will require a custom mailbox implementation).

Answer 2

If it's possible with your algorithm another approach would be to turn the thing around and have nodes ask other random nodes for data at random points in time. Work pulling is usually easier to control than pushing.