Problems with node tool repair

Question

The environment

We have following data centers of Cassandra cluster (version 2.0.5) running on Red Hat 6.5:

• DC1: 5 nodes with NetworkTopologyStrategy set to 5
• DC2: 4 nodes with NetworkTopologyStrategy set to 3

The script creating KEYSPACE is following:

CREATE KEYSPACE OUR_STUFF WITH replication = {'class':'NetworkTopologyStrategy', 'DC1':5, 'DC2':3};

We don’t have any other keyspace (except of course of system_auth)

We write always to one DC using LOCAL_QUORUM option.

We have network connection problem between DC1 and DC2, they happened every day, theoretically the connection can be broke for days/weeks.

For test proposes we load into cluster ~50 GB of data.

The problem

The main problem is that command: node tool repair hangs usually and can took over few hours - with par option more or less 3 hours, without par option 6 hours. Moreover in last 2 weeks node tool repair command hangs and nothing happened for more than one day. We don’t find any ERROR in log files. We don’t load more data into cluster. It is suggested by Cassandra to run node tool repair every 10 day on every node, and that you shouldn’t run more than one repair. But we cannot find answer to following question:

• How to check status of repair command?
• What we should do if repair command hangs?
• For what we should look in log files to check what is wrong?
• How to avoid running 2 repair commands in parallel?
• And the most important one (which is sum on above): how to avoid problems with nodetool repair command?

Answer 1

How to check status of repair command?

Looking at the logs (and the output of the repair command itself) will tell you the status of the repair in terms of what is going on. nodetool netstats and nodetool compactionstats will also tell you what operations are pending.

What we should do if repair command hangs?

Restarting one of the nodes is the easiest way to clear the repair.

For what we should look in log files to check what is wrong?

Look at the repair session logs and see what sessions started, and messages have been sent between the nodes. If you have "network issues", most likely those are causing your problem. You will most likely want to set the streaming_socket_timeout_in_ms to have repair streams restart themselves if the network hiccups https://issues.apache.org/jira/browse/CASSANDRA-3838

How to avoid running 2 repair commands in parallel?

You can use JMX to check if there are already repairs running.

And the most important one (which is sum on above): how to avoid problems with nodetool repair command?

Fix your network :/ If your network is unreliable, repair is going to also be unreliable, as it does need to use the network...