Creating output file according to date in oozie
Question
I am using oozie to run my map-reduce job. I want to create the output file according to the date. But it takes date as a string and ends up printing instead of taking date as the value :
/user/skataria/geooutput/$(date +"%m%d%Y%H%M%S")
Here is the oozie properties file:
nameNode=hdfs://localhost:8020
jobTracker=localhost:8021
date=(date +"%m%d%Y%H%M%S")
oozie.wf.application.path=${nameNode}/services/advert/sid
inputDir=${nameNode}/user/${user.name}/geoinput/testgeo
outputDir=${nameNode}/user/${user.name}/geooutput/${date}
Somehow i cant have oozie as tag because my reputation is under 1500
Solution
It looks like you're trying to use a linux shell command (date +"%m%d%Y%H%M%S"
) in a java properties file - this isn't going to resolve.
One work around, assuming this is part of a manually submitted Workflow job (as opposed to a Coordinator job) is to provide the date property from the command line using the -D key=value option, and linux shell back quotes to resolve the output of a command inline
oozie job -run -config job.properties -D date=`date +"%m%d%Y%H%M%S"`
You'll need to make sure your version of Oozie support the -D key=value option
OTHER TIPS
Yes I agree the shell option works. But that does not solve my usecase. I want to run my map-reduce job daily and schedule this thru Hue. The output directory needs to be parameterized as an job property to Oozie.
By the way I find that Oozie has Expression language Functions,
Unfortunately the function timestamp()
returns the UTC current date and time in W3C format down to the second (YYYY-MM-DDThh:mm:ss.sZ)
. i.e.: 1997-07-16T19:20:30.45Z
and completely unusable for creating a sub-directory name in HDFS
So for now,
I have a workaround. I am using the Workflow EL Function wf:id()
In workflow.xml
<property>
<name>mapreduce.output.fileoutputformat.outputdir</name>
<value>/user/sasubramanian/impressions/output/outpdir/${yyyy_mm_dd}/${wf:id()}</value>
</property>
This creates a output directory with subdirectory as,
/user/foouser/subdir1/output/outpdir/0000006-130321100837625-oozie-oozi-W
NOTE: You must specify this in the workflow.xml. This will not work if you specified it in the job.properties