Moving a node to a different machine

Question

We have the following DSE cluster setup:

DC Cassandra

• Cassandra node 1

DC Solr

• Solr node 1
• Solr node 2
• Solr node 3
• Solr node 4

We want to replace Solr node 1 with a more powerful machine. I'm under the impression that we need to follow the procedure for replacing a dead node which involves:

• Adding the new node to the cluster
• Allowing the cluster to automatically re-balance itself
• Removing the old node via nodetool removenode
• Running nodetool cleanup in each remaining node

However, my colleage resorted to copying everything (user files, system files, and the Cassandra/Solr data files) from the old machine to the new machine. Will this approach work? If yes, is there any additional step that we need to do? If not, how can we correct this? (i.e. do we simply delete the datafiles and restart the node as an empty node? or will doing so lead to data loss?)

Answer 1

So your approach should work... here are some things to observe

• Make sure you shut down your C* on the node to replace.
• Make it impossible to start C* on the old node by accident (move the jar files away for example, or at least temporarily move the /etc/init.d/dse script somewhere else)
• Copy everything to the new machine
• Shutdown the old machine (disconnect network if possible).
• Make sure that the new machine has the same ip address as the old one, and that for the first boot it's not gonna start C* (not a real requirement, but more a precaution in case the ip address doesn't match, or there is something else wrong with that box).
• double check everything is fine, reenable C* and restart the machine. Depending on how you copied that machine I would be more concerned with OS system files in terms of stability. If you just copied the C* app and data files you should be fine.
• make sure you NEVER start the old machine with an active C*.

I haven't tried this, but there isn't anything I know off that would prevent this from working (now that I said this, I am probably gonna get dinged... but I DID ask one of our key engineers :-).

The more "standard" procedure is this, which I will propose for our docs:

Replacing a running node

Replace a node with a new node, for example to update to newer hardware/proactiv e maintenance.

You must prepare and start the replacement node, integrate it into the cluster, and then remove the old node.

Procedure

1. Confirm that the node is alive:

   a) Run nodetool ring if not using vnodes.

   b) Run nodetool status if using vnodes.

2. The nodetool command shows a up status for the node (UN)

3. Note Host ID of the node to replace; it is used in the last step.
4. Add and start the replacement node as described in http://www.datastax.com/docs/1.1/cluster_management#adding-capacity-to-an-existing-cluster
5. Using the Host ID of the original old node, remove the old node from the cluster using the nodetool removenode command. See http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_remove_node_t.html for detaile dinstructions.