How does a VoltDB cluster determine time sync between nodes

Question

I have a weird painful issue with VoltDB clusters on multiple RHEL7.1 vm (KVM).

Time is synced between the VM and their hypervisor with a single local master NTP server - using chrony. Master server sync with external ntp pool.

There is a random error generated by VoltDB:

FATAL [main] HOST: Clock skew is 120 which is > than the 100 millisecond limit. Make sure NTP is running.

Chrony config has been setup as much as possible as per the perf guide from VoltDB for ntpd - here.

I have searched high and low with my "google-fu" - but I can't seem to find an answer on how VoltDB determines that clock skew error.

The only closest one I can find is in this java code: voltdb/SocketJoiner.java on github from line 603 onwards.

skew = System.currentTimeMillis() - currentTimeBuf.getLong();

I wonder why VoltDB doesn't use a java function call - System.nanoTime() instead, for a precision timing.

Any help or insights will be great.

Answer 1

At startup, VoltDB is going to send a round-trip message from the first node of the cluster to all joining nodes of the cluster. This message queries the local time at each node.

For each joining node, the difference between its local time and the first node's local time is computed. This includes clock skew and network latency. In fact, it's possible for clock skew and network latency to cancel each other out, but not super likely.

Across all joining nodes, VoltDB reports the maximum difference between clocks.

In this case, VoltDB is telling you that for some pair of nodes including a joining node and the first node you started (the startup leader), the difference between the compared clocks was 120ms, which is a lot for machines on NTP that are in the same datacenter.

You can check network latency using ping between nodes, but make sure you're using the same network interface you've setup VoltDB to run on.