Datanode not starts correctly

Question

I am trying to install Hadoop 2.2.0 in pseudo-distributed mode. While I am trying to start the datanode services it is showing the following error, can anyone please tell how to resolve this?

**2**014-03-11 08:48:15,916 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (storage id unknown) service to localhost/127.0.0.1:9000 starting to offer service
2014-03-11 08:48:15,922 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-03-11 08:48:15,922 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2014-03-11 08:48:16,406 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode/in_use.lock acquired by nodename 3627@prassanna-Studio-1558
2014-03-11 08:48:16,426 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-611836968-127.0.1.1-1394507838610 (storage id DS-1960076343-127.0.1.1-50010-1394127604582) service to localhost/127.0.0.1:9000
java.io.IOException: Incompatible clusterIDs in /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode: namenode clusterID = CID-fb61aa70-4b15-470e-a1d0-12653e357a10; datanode clusterID = CID-8bf63244-0510-4db6-a949-8f74b50f2be9
    at**** org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:391)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:191)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:219)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:837)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:808)
    at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:280)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:222)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
    at java.lang.Thread.run(Thread.java:662)
2014-03-11 08:48:16,427 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-611836968-127.0.1.1-1394507838610 (storage id DS-1960076343-127.0.1.1-50010-1394127604582) service to localhost/127.0.0.1:9000
2014-03-11 08:48:16,532 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-611836968-127.0.1.1-1394507838610 (storage id DS-1960076343-127.0.1.1-50010-1394127604582)
2014-03-11 08:48:18,532 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2014-03-11 08:48:18,534 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2014-03-11 08:48:18,536 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 

Answer 1

You can do the following method,

copy to clipboard datanode clusterID for your example, CID-8bf63244-0510-4db6-a949-8f74b50f2be9

and run following command under HADOOP_HOME/bin directory

./hdfs namenode -format -clusterId CID-8bf63244-0510-4db6-a949-8f74b50f2be9

then this code formatted the namenode with datanode cluster ids.

Answer 2

You must do as follow :

• bin/stop-all.sh
• rm -Rf /home/prassanna/usr/local/hadoop/yarn_data/hdfs/*
• bin/hadoop namenode -format

I had the same problem until I found an answer in this web site.

Answer 3

Whenever you are getting below error, trying to start a DN on a slave machine:

java.io.IOException: Incompatible clusterIDs in /home/hadoop/dfs/data: namenode clusterID= ****; datanode clusterID = ****

It is because after you set up your cluster, you, for whatever reason, decided to reformat your NN. Your DNs on slaves still bear reference to the old NN.

To resolve this simply delete and recreate data folder on that machine in local Linux FS, namely /home/hadoop/dfs/data.

Restarting that DN's daemon on that machine will recreate data/ folder's content and resolve the problem.

Answer 4

Do following simple steps

• Clear the data directory of hadoop
• Format the namenode again
• start the cluster

After this your cluster will start normally if you are not having any other configuration issue

Answer 5

DataNode dies because of incompatible Clusterids compared to the NameNode. To fix this problem you need to delete the directory /tmp/hadoop-[user]/hdfs/data and restart hadoop.

rm -r /tmp/hadoop-[user]/hdfs/data

Answer 6

I got similar issue in my pseudo distributed environment. I stopped cluster first, then I copied Cluster ID from NameNode's version file and put it in DataNode's version file, then after restarting cluster, its all fine.

my data path is here /usr/local/hadoop/hadoop_store/hdfs/datanode and /usr/local/hadoop/hadoop_store/hdfs/namenode.

FYI : version file is under /usr/local/hadoop/hadoop_store/hdfs/datanode/current/ ; likewise for NameNode.

Answer 7

Here, the datanode gets stopped immediately because the clusterID of datanode and namenode are different. So you have to format the clusterID of namenode with clusterID of datanode

Copy the datanode clusterID for your example, CID-8bf63244-0510-4db6-a949-8f74b50f2be9 and run following command from your home directory. You can go to your home dir by just typing cd on your terminal.

From your home dir now type the command:

hdfs namenode -format -clusterId CID-8bf63244-0510-4db6-a949-8f74b50f2be9

Answer 8

Delete the namenode and datanode directories as specified in the core-site.xml. After that create the new directories and restart the dfs and yarn.

Answer 9

I also had the similar issue. I deleted namenode and datanode folders from all the nodes, and rerun:

$HADOOP_HOME/bin> hdfs namenode -format -force
$HADOOP_HOME/sbin> ./start-dfs.sh
$HADOOP_HOME/sbin> ./start-yarn.sh

To check the health report from command line (which I would recommend)

$HADOOP_HOME/bin> hdfs dfsadmin -report

and I got all the nodes working correctly.

Answer 10

I had same issue for hadoop 2.7.7

I removed the namenode/current & datanode/current directory on namenode and all the datanodes

• Removed files at /tmp/hadoop-ubuntu/*
• then format namenode & datanode
• restart all the nodes.
• things work fine

steps: stop all nodes/managers then attempt below steps

1. rm -rf /tmp/hadoop-ubuntu/* (all nodes)
2. rm -r /usr/local/hadoop/data/hdfs/namenode/current (namenode: check hdfs-site.xml for path)
3. rm -r /usr/local/hadoop/data/hdfs/datanode/current (datanode:check hdfs-site.xml for path)
4. hdfs namenode -format (on namenode)
5. hdfs datanode -format (on namenode)
6. Reboot namenode & data nodes

Answer 11

There's been different solutions to this problem, but I tested another easy solution and it worked like a charm :

So if someone get the same error, you just need to change the clusterID in the datanodes with clusterID of the namenode in the VERSION file.

With your case, here's were you can change it on datanode side :

namenode clusterID = CID-fb61aa70-4b15-470e-a1d0-12653e357a10; datanode clusterID = CID-8bf63244-0510-4db6-a949-8f74b50f2be9

• Backup the current VERSION : cp /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode/current/VERSION /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode/current/VERSION.BK
• vim /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode/current/VERSION and change

clusterID=CID-8bf63244-0510-4db6-a949-8f74b50f2be9

with

clusterID=CID-fb61aa70-4b15-470e-a1d0-12653e357a10

Restart the datanode and it should work.