cassandra opscenter phantom node

Question

I have had a 6 node Cassandra (v 2.03) running for a few weeks. I tried to add a new node yesterday, but had a streaming error.

I found the cause of this and thought I would decommission the new node, fix the problem, the go back to adding the new node.

I removed the new node, and when I run nodetool status, only 6 nodes show, the cluster was up and still running with no issues.

However, in opscenter, it shows 7 nodes, even though this machine is now terminated, it still thinks it is part of the cluster.

I have now uninstalled the opscenter agent on all machines, removed opscenter, all logs, and dropped the opscenter keyspace. When I reinstalled opscenter, the 7th node was still there.

As mentioned, this machine does not exist anymore, and it is not shown in nodetool status, so i cannot issue any commands related to this machine.

Could someone please tell me, where is opscenter receiving this information from?

I deleted everything to do with opscenter, and nodetool thinks everything is ok, so something somewhere is holding on to info about this dead node, but I do not know where it is and how to remove it. thanks

nodetool status output :

http://pastebin.com/HFAkVmLY

you can see it shows 6 nodes, but opscenter has 7

Answer 1

I fixed this problem by using the following query in cqlsh

delete from system.peers where peer = 'ipaddress_you_want_to_remove';