Some cassandra nodes in cluster are unreachable even all the nodes are up and running

Question

I have cassandra cluster of 6 nodes on ec2 , Many times i have observed that all the nodes in the cluster are up and running but when i do the describe cluster from client it shows some nodes as unreachable.

1. I can login to individual machines

2. can connect to cassandra node with cassandra client

3. can connect to individual nodes with JMX.

Answer 1

sounds like you have listen_address/broadcast_address misconfigured.

Answer 2

I' am using hector as Cassandra client - maybe you have similar problem.

On CassandraHost you can configure ExhaustedPolicy. Default, if Cassandra is busy, such host will be marked as down - even if it is online. Hector checks all hosts marked as down every retryDownedHostsDelayInSeconds.

Is is possible, that your host was busy for some time, it was marked down, and retry was not triggered yet?