How do you backup Zookeeper?

Question

I have read that copying the data directory will work. But, that is a combination of logs and snapshots. How do folks backup a zookeeper infrastructure ? Export ? Dump ? Custom script ? What are the best practices ?

Answer 1

We're modifying the zkConfig.py script which is a contributed project when you install zookeeper. It lets you interact with zookeeper through a python script.

We're modifying the scripts to easily dump and save the entire contents each night and then backup the files. Though I would be curious to hear other people's solutions to this as well.

Answer 2

Zookeeper writes a snapshot once it determines that it has enough transactions and every new snapshot completely supersedes older ones. So the latest snapshot + the transaction log from the time of the snapshot is enough to recover to current state. To make the calculations easier, you can simply backup the last 3 snapshots(in case of corruption of the latest snap) and the transaction logs from the timestamp corresponding to the earliest snapshot. The links below have some more details.

1. http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_dataFileManagement
2. http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

Answer 3

Netflix provided a solution for this called exhibitor. It's a "ZooKeeper co-process for instance monitoring, backup/recovery, cleanup and visualization."

Answer 4

I just had the same requirement and found that most of the available options either don't work or require a lot of customisation.

The best option I found was Guano which is a little Java app that visits each node in the tree recursively starting from the declared node and dumps it into a matching directory structure, so you end up with a directory structure of plain files that is structured like the actual tree.

You can also restore these backups by asking it to restore recursively from any point in that tree. I think this is quite nice both for backups and for exploration. For example I immediately used ack from the root to find all files with an entry I cared about.

This is easy to extend into a proper backup simply by putting it as a cron job and adding a zip step to compress the whole backup into an archive as well as handle any rotation needed.

There are a few downsides to the tool:

1. As it stands on Github the original does not compile due to missing a few imports. Several people have made PRs or forks that fix this issue such as https://github.com/feldoh/guano which is my fork wherein I also improved the docs. I have also now pre-compiled the jar and will be pushing binaries into https://bintray.com/feldoh/Guano/guano.
2. It dumps the data only, which is good for exploration but loses meta-data such as the mTime or the data version. Admittedly a restore probably should count as an update so I can't say its really a bad thing, but its not a true point-in-time restore.

NB: I have made my own Zookeeper editor as I had similar problems finding one of those that worked and met my needs. Depending on when you read this https://github.com/feldoh/JZookeeperEdit may also have an export feature. Issues 13/14 cover this planned feature.

Answer 5

There's a very nice tool called zk-shell that can do an enormous amount of things with Zookeeper. It has a mirror command that can copy an entire Zookeeper tree recursively to/from Zookeeper or local JSON file.

Source & documentation: https://github.com/rgs1/zk_shell

Installation on Centos 7:

yum install python2-pip
pip install zk_shell

Example to back up a zookeeper tree to a local JSON file:

zk-shell localhost:2181 --run-once 'mirror / json://!tmp!zookeeper-backup.json/'