Network protocol for multi-way communication

Question

I'm writing a piece of software that needs to be run on multiple machines (3-4 at a time), and show the same data. Data can be modified from each one of the machines, and the changes would be immediately sent to the other machines.

There is no central server, but all machines are friendly and on the same local network.

The most important things here are (1) low latency and (2) serializability of messages (always being able to know which came first) - basically what I'm looking for me a chatroom, but for the programs' messages. What's the best network protocol to use for that?

Answer 1

In layer four you would use TCP for reliable delivery of messages. If you are talking about 3-4 machines that should be OK. Above TCP you could use your own application-specific protocol. I don't know if XMPP would let you work in a de-centralized environment.

One problem you'll need to solve is that all see the messages in the same sequence. For that you can use vector clocks algorithm or some way to solve conflicts in case two messages are sent at the same time.

In a LAN any protocol should be fine for low latency.

Regarding the number of connections you will need, that will depend on your design. Always thinking of a low number of peers:

1. For a mesh kind of connectivity, you need TCP connections in both directions for every 2 peers.
2. You could adopt another design: circular connectivity, where each node is connected to one neighbor. That would reduce number of existing connections with different challenges (what happens if one neighbor crashes or shuts down).
3. Other interesting alternative that I implemented some years ago: Use UDP multicast to send the message to all members of the multicast group, then all receivers open a TCP connection with the sender to send back the ACK (acknowledgement). The sender then checks all ACK's received and if there is anyone missing, it opens a TCP connection with that one and sends the message reliably.

Something else you must consider: group management. How do you know one peer left the group or a new peer joins the group? That is another consideration.