I have some problems when I test hadoop multi node cluster

StackOverflow https://stackoverflow.com/questions/23294665

Question

Hi I'm using hadoop 0.23.9 with 2 node when I tried the singel node it works fine but when i try to test word count job I get the following result and I don't knew where the error

I use ubuntu 12.04 LTS in both PCs

        mapred-site.xml


        <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
        </property>

        </configuration>

         yarn-site.xml


<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce.shuffle</value>
  </property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>


 <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
  </property>

<property>
 <name>mapreduce.job.tracker</name>
  <value>master</value>
 </property>
<configuration>

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce.shuffle</value>
  </property>




<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>

<property>
<name>yarn.nodemanager.address</name>
<value>0.0.0.0:8042</value>
</property>

</configuration>

and here the hosts file of ubuntu 

    127.0.0.1       localhost
    127.0.1.1       fatima1-HP-ProBook-4530s

192.168.1.10    master
192.168.1.11    slave

#the following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters









        14/04/25 15:41:39 INFO input.FileInputFormat: Total input paths to process : 7
        14/04/25 15:41:39 WARN snappy.LoadSnappy: Snappy native library not loaded
        14/04/25 15:41:39 INFO mapreduce.JobSubmitter: number of splits:7
        14/04/25 15:41:39 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.value.class is deprecated.    Instead, use mapreduce.job.output.value.class
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
        14/04/25 15:41:39 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
        14/04/25 15:41:39 INFO mapred.ResourceMgrDelegate: Submitted application application_1398429615167_0001 to ResourceManager at master/192.168.1.10:8032
        14/04/25 15:41:39 INFO mapreduce.Job: The url to track the job: `http://master:8088/proxy/application_1398429615167_0001/`
        14/04/25 15:41:39 INFO mapreduce.Job: Running job: job_1398429615167_0001
        14/04/25 15:41:46 INFO mapreduce.Job: Job job_1398429615167_0001 running in uber mode : false
        14/04/25 15:41:46 INFO mapreduce.Job:  map 0% reduce 0%
        14/04/25 15:41:47 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_0, Status : FAILED
    Container launch failed for container_1398429615167_0001_01_000008 : java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
    Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
        at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
        at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:49 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_1, Status : FAILED
Container launch failed for container_1398429615167_0001_01_000009 : java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
    at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:51 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_2, Status : FAILED
Container launch failed for container_1398429615167_0001_01_000010 : java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
    at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:53 INFO mapreduce.Job:  map 14% reduce 0%
14/04/25 15:41:53 INFO mapreduce.Job: Job job_1398429615167_0001 failed with state FAILED due to: Task failed task_1398429615167_0001_m_000006
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/04/25 15:41:53 INFO mapreduce.Job: Counters: 5
    Job Counters 
        Launched map tasks=6
        Other local map tasks=3
        Data-local map tasks=7
        Total time spent by all maps in occupied slots (ms)=0
        Total time spent by all reduces in occupied slots (ms)=0
Was it helpful?

Solution 2

After trying every possible solution to the problem, the solution was: making the host name of the PC that work as master "master" and the slave "slave". After that entering the ip addresses of the PCs in the file "/etc/hosts".

Thanks a lot to Mr. apesa for his answers.

OTHER TIPS

Your hostname is not set correctly in /etc/hosts. You should remove the following line altogether.

127.0.1.1       fatima1-HP-ProBook-4530s

Then add your actual IP address right after the localhost entry. Your /etc/hosts file should look like this. NOTE: I am assuming this is the master, I may be wrong in which case replace it wiuth the actual IP address. Also check your destination machines /etc/hosts file as well.

    127.0.0.1       localhost
    192.168.1.10    fatima1-HP-ProBook-4530s

192.168.1.10    master
192.168.1.11    slave

the following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Once this has been completed you will need to reboot your machine and restart hadoop

EDIT: Your core-site.xml has to have the correct master / slave entries as follows. You may have additional properties set within core-site.xml and that is fine. But in Distributed Mode you will need to specify a fs.default.name as a master or a slave.

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://master:9000</value>
  </property>
</configuration>
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top