Multinode Hadoop slave nodes cannot access a jar file on the master node

Question

I'm using streaming to call a jar file to perform some tasks, as:

hadoop jar /path/to/hadoop-streaming.jar -input /inDir -ouput /outDir -file jarscript.sh -mapper jarscript.sh

where jarscript.sh is:

java -jar /path/to/jar/X.jar -arguments

When I run the streaming command, it works fine on the master node, except that I get errors on the slave nodes that they cannot access X.jar. How do I change that? How can I enable access to the jar file on the slave nodes? Is there a specific location they need to be in order for the slave nodes to access the jar file?

I used Michael Noll's tutorial, so hadoop is on hduser and the jar file is in another user's space - hadoopmaster, so that path is more like /home/hadoopmaster/path/to/jar/X.jar. Could this be an issue?

So, I tried Donald's way of doing it and still the slaves give me this error:

stderr logs

Unable to access jarfile /home/hadoopmaster/Downloads/PaDEL-Descriptor/PaDEL-Descriptor.jar
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)



syslog logs

2013-12-09 11:18:13,183 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2013-12-09 11:18:13,338 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/jars/META-INF <- /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/attempt_201312091116_0001_m_000000_1/work/META-INF
2013-12-09 11:18:13,351 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/jars/jarscript.sh <- /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/attempt_201312091116_0001_m_000000_1/work/jarscript.sh
2013-12-09 11:18:13,359 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/jars/org <- /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/attempt_201312091116_0001_m_000000_1/work/org
2013-12-09 11:18:13,371 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/jars/lib <- /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/attempt_201312091116_0001_m_000000_1/work/lib
2013-12-09 11:18:13,386 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/jars/.job.jar.crc <- /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/attempt_201312091116_0001_m_000000_1/work/.job.jar.crc
2013-12-09 11:18:13,387 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/jars/job.jar <- /app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/attempt_201312091116_0001_m_000000_1/work/job.jar
2013-12-09 11:18:13,598 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-12-09 11:18:13,691 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-12-09 11:18:13,695 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@31598bd
2013-12-09 11:18:13,770 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoopmaster:54310/BiolData/input/biolink1/compound_id_464726_2d_3D.sdf:0+13142
2013-12-09 11:18:13,785 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-12-09 11:18:13,792 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2013-12-09 11:18:13,800 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
2013-12-09 11:18:13,840 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720
2013-12-09 11:18:13,840 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
2013-12-09 11:18:13,854 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201312091116_0001/attempt_201312091116_0001_m_000000_1/work/./jarscript.sh]
2013-12-09 11:18:13,873 INFO org.apache.hadoop.streaming.PipeMapRed: MRErrorThread done
2013-12-09 11:18:13,877 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
2013-12-09 11:18:13,878 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]
2013-12-09 11:18:13,879 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]
2013-12-09 11:18:13,883 WARN org.apache.hadoop.streaming.PipeMapRed: java.io.IOException: Stream closed
at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:434)
at java.io.OutputStream.write(OutputStream.java:116)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:569)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

2013-12-09 11:18:13,883 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed failed!
2013-12-09 11:18:13,913 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-12-09 11:18:13,947 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2013-12-09 11:18:13,947 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName hduser for UID 1001 from the native implementation
2013-12-09 11:18:13,950 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-12-09 11:18:13,953 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

Answer 1

The problem is that your X.jar probably isn't on every node already... right? So you need to ship it everywhere when the job runs.

You can add X.jar to the -file parameter, like this:

hadoop jar /path/to/hadoop-streaming.jar
       -input /inDir -ouput /outDir
       -file jarscript.sh
       -file /path/to/jar/X.jar
       -mapper jarscript.sh

This ships that jar file to every node when you run the job (just like how it ships your jarscript.sh).

Note that you shouldn't use an absolute path in the shell script anymore. X.jar will be in the current working directory of the shell script, so you should change your script to java -jar X.jar -arguments.

Answer 2

How about something like this?

First, before starting the job, put the jar in some known location in HDFS, e.g.:

hadoop fs -put /path/to/jar/X.jar /lib

Then add a line to your jarscript.sh to first copy that jar out of HDFS, like this:

hadoop fs -get /lib/X.jar .
java -jar ./X.jar -arguments

It's a bit of a hack, but I think it should work if you can't get the "right" way with the "-file" argument that Donald suggested to work.