ZooKeeper linearizability regarding watched notification

Question

Question:

If client A is watching znode /a and is trying to read /b; and client B deletes /a before updating /b. Client A will stop reading /b if it gets the notification that /a is gone.

Is it possible for the following to occur in order?

1. client A reads /a and sets watch
2. client A sees /a exists
3. client B deletes /a
4. client B receives response that /a is gone
5. client A reads /b
6. client A gets outdated information
7. client A gets notified /a is gone, but it's too late

Answer 1

I assume there is a 3.5 Client B updates /b, also I don't think 4. is relevant.

The zookeeper guarantees are here,

Watches are ordered with respect to other events, other watches, and asynchronous replies. The ZooKeeper client libraries ensures that everything is dispatched in order.

If you use java, and only use the async methods, then in the zookeeper event thread, you will get a watchEvent that /a was deleted before the async read of /b.

There is a complication though if you use the synchronous zookeeper api, since then you are introducing more threads in your code which can violate the ordering guarantees. See the notes in the java bindings here, especially this part,

Synchronous calls may not return in the correct order. For example, assume a client does the following processing: issues an asynchronous read of node /a with watch set to true, and then in the completion callback of the read it does a synchronous read of /a. (Maybe not good practice, but not illegal either, and it makes for a simple example.) Note that if there is a change to /a between the asynchronous read and the synchronous read, the client library will receive the watch event saying /a changed before the response for the synchronous read, but because the completion callback is blocking the event queue, the synchronous read will return with the new value of /a before the watch event is processed.

So if you are using the synchronous api A may see the watch event of /a after the read of /b.

Answer 2

In zookeeper only write obey linearizability not read. Because of that event when using Async API, its possible in step 5 for a client to read a stale value of A after some client got a response that A was succesfully deleted.

This is because your read request might reach a Zookeeper node that think its still the master node and this node will not try to talk to other replica before responding to the read request so it would not discover it's no longer master fast enough.

If you want your read to return the latest value you can force linearizability by always calling Sync() before doing a read.

This would guarantee that your read will see the effect of all write that had happen in wall clock time, before your call to Sync().