Files not stored in Distributed Cache

Question

I am using DistributedCache. But there are no files in the cache after execution of code. I have referred to other similar questions but the answers does not solve my issue.

Please find the code below:

   Configuration conf = new Configuration();
   Job job1 = new Job(conf, "distributed cache");
   Configuration conf1 = job1.getConfiguration();
   DistributedCache.addCacheFile(new Path("File").toUri(), conf1);
   System.out.println("distributed cache file "+DistributedCache.getLocalCacheFiles(conf1));

This gives null..

The same thing when given inside mapper also gives null hence. Please let me know your suggestions.

Thanks

Answer 1

try getCacheFiles() instead of getLocalCacheFiles()

Answer 2

I believe this is (at least partly) due to what Chris White wrote here:

After you create your Job object, you need to pull back the Configuration object as Job makes a copy of it, and configuring values in conf2 after you create the job will have no effect on the job iteself. Try this:

job = new Job(new Configuration());
Configuration conf2 = job.getConfiguration();
job.setJobName("Join with Cache");
DistributedCache.addCacheFile(new URI("hdfs://server:port/FilePath/part-r-00000"), conf2);

I guess if it still does not work, there is another problem somewhere, but that doesn't mean that Chris White's point is not correct.

Answer 3

When distributing, don't forget the local link name, preferably using a relative path:

URI is of the form hdfs://host:port/absolute-path#local-link-name

When reading:

• if you don't use distributed cache possibilities, you are supposed to use HDFS's FileSystem to access the hdfs://host:port/absolute-path
• if you use the distributed cache, then you have to use standard Java file utilities to access the local-link-name

Answer 4

The cache file needs to be in the Hadoop FileSystem. You can do this: void copyFileToHDFS(JobConf jobConf, String from, String to){

    try {
        FileSystem aFS = FileSystem.get(jobConf);
        aFS.copyFromLocalFile(false, true, new Path(
                from), new Path(to));
    } catch (IOException e) {
        throw new RuntimeException(e);
    } 
}

Once the files are copied you can add them to the cache, like so:

    void fillCache(JobConf jobConf){
        Job job;
        copyFileToHDFS(jobConf, fromLocation, toLocation);
        job = Job.getInstance(jobConf);
        job.addCacheFile(new URI(toLocation));
        JobConf newJobConf = new JobConf(job.getConfiguration());
    }