how to set jar path when submit a remote application to hadoop?

Question

I try to run the first example from at website http://hadoop.apache.org/docs/current2/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html

the follow's is my code

    Path jarPath = new Path("target/HadoopStudy-0.0.1-SNAPSHOT.jar");
    FileSystem fs = FileSystem.get(conf);
    FileStatus jarStatus = fs.getFileStatus(jarPath);
    LocalResource amJarRsrc = Records.newRecord(LocalResource.class);

    amJarRsrc.setType(LocalResourceType.FILE);

    amJarRsrc.setVisibility(LocalResourceVisibility.APPLICATION);


    logger.debug(ConverterUtils.getYarnUrlFromPath(jarPath));
    amJarRsrc.setResource(ConverterUtils.getYarnUrlFromPath(jarPath));
    amJarRsrc.setTimestamp(jarStatus.getModificationTime());
    amJarRsrc.setSize(jarStatus.getLen());
    localResources.put(Common.LOCAL_JAR_PATH, amJarRsrc);
    amContainer.setLocalResources(localResources);

but I can get the application id. and submit the application to yarn,but the application will always failed,I check the nodemanger log at the master, I found these error

java.net.URISyntaxException: Relative path in absolute URI: ://target/HadoopStudy-0.0.1-SNAPSHOT.jar
    at java.net.URI.checkPath(URI.java:1804)
    at java.net.URI.<init>(URI.java:752)
    at org.apache.hadoop.yarn.util.ConverterUtils.getPathFromYarnURL(ConverterUtils.java:77)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.<init>(LocalResourceRequest.java:46)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:523)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:494)
    at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
    at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
    at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
    at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:850)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:73)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:681)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:674)
    at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
    at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:81)
    at java.lang.Thread.run(Thread.java:722)

I run the example on PC A,but I deploy hadoop at PC B. and the jar is at PC A.

after investigate, i can't fix it.but I have some question.
1) do I need upload the jar to hdfs? if i need, I know what to do
2) I just to understand some principle of yarn, if i want to submit an application, I need to upload the jar to master? Can I upload the jar file to master's local file system? if I can, how to set path?

Answer 1

1. do I need upload the jar to hdfs? if i need, I know what to do

   Yes.

2. I just to understand some principle of yarn, if i want to submit an application, I need to upload the jar to master? Can I upload the jar file to master's local file system? if I can, how to set path?

   YARN NodeManagers use a FileSystem interface to 'localize' jars on to the local nodes. For that to work, you need to upload all your resource upfront to a central place accessible via a FileSystem implementation, typically HDFS.

   So, if you have a path "target/HadoopStudy-0.0.1-SNAPSHOT.jar", first upload it to your home directory on HDFS, let's say hdfs:///user/song/HadoopStudy-0.0.1-SNAPSHOT.jar" and then set that path in the local-resource.