Error on starting HDFS daemons on hadoop Multinode cluster

Question

Issue While Hadoop multi-node set-up .As soon as i start My hdfs demon on Master (bin/start-dfs.sh)

i did got below logs on Master

starting namenode, logging to /home/hduser/hadoop/libexec/../logs/hadoop-hduser-namenode-localhost.localdomain.out
slave: Warning: $HADOOP_HOME is deprecated.
slave:
slave: starting datanode, logging to /home/hduser/hadoop/libexec/../logs/hadoop-hduser-datanode-localhost.localdomain.out
master: Warning: $HADOOP_HOME is deprecated.
master:
master: starting datanode, logging to /home/hduser/hadoop/libexec/../logs/hadoop-hduser-datanode-localhost.localdomain.out
master: Warning: $HADOOP_HOME is deprecated.
master:
master: starting secondarynamenode, logging to /home/hduser/hadoop/libexec/../logs/hadoop-hduser-secondarynamenode-localhost.localdomain.out

i did got below logs on slave @

hadoop-hduser-datanode-localhost.localdomain.log file

can some advise me , whats the wrong with set-up .

2013-07-24 12:10:59,373 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.1:54310. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2013-07-24 12:11:00,374 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.1:54310. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2013-07-24 12:11:00,377 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to master/192.168.0.1:54310 failed on local exception: java.net.NoRouteToHostException: No route to host
        at org.apache.hadoop.ipc.Client.wrapException(Client.java:1144)
        at org.apache.hadoop.ipc.Client.call(Client.java:1112)

Answer 1

Make sure your NameNode is running fine. If it is already running see if there is any problem in the connection. Your DataNode is not able to talk to the NameNode. Make sure you have added the IP and hostname of the machine in the /etc/hosts file of your slave. Try telnet to 192.168.0.1:54310 and see whether you are able to connect or not.

Showing us the NN logs would be helpful.

Edit :

See what the wiki has to say about this problem : You get a TCP No Route To Host Error -often wrapped in a Java IOException, when one machine on the network does not know how to send TCP packets to the machine specified.

Some possible causes (not an exclusive list):

• The hostname of the remote machine is wrong in the configuration files.
• The client's host table //etc/hosts has an invalid IPAddress for the target host.
• The DNS server's host table has an invalid IPAddress for the target host.
• The client's routing tables (In Linux, iptables) are wrong.
• The DHCP server is publishing bad routing information.
• Client and server are on different subnets, and are not set up to talk to each other. This may be an accident, or it is to deliberately lock down the Hadoop cluster.
• The machines are trying to communicate using IPv6. Hadoop does not currently support IPv6
• The host's IP address has changed but a long-lived JVM is caching the old value. This is a known problem with JVMs (search for "java negative DNS caching" for the details and solutions).

The quick solution: restart the JVMs.

These are all network configuration/router issues. As it is your network, only you can find out and track down the problem.