How can to get the filename from a streaming mapreduce job in R?

Question

I am streaming an R mapreduce job and I am need to get the filename. I know that Hadoop sets environment variables for the current job before it starts and I can access env vars in R with Sys.getenv().

I found : Get input file name in streaming hadoop program

and Sys.getenv(mapred_job_id) works fine, but it is not what I need. I just need the filename and not the job id or name. I also found: How to get filename when running mapreduce job on EC2?

But this isn't helpful either. What is the easiest way to get the current filename while streaming from R? Thank you

Answer 1

I have not tried this, but from the second link you provided, it seems that this is available in an environment variable called map.input.file. Then, this should work:

Sys.getenv("map.input.file")

EDIT: Upon further investigation, I learned that you need to replace the dots with underscores, so this is the way to do it:

Sys.getenv("map_input_file")

However, the map.input.file property has been deprecated in YARN (Hadoop 2.x), so the new name should be used instead:

Sys.getenv("mapreduce_map_input_file")