Talend How To Pass Last Modified File Into TFileInputDelimited?

Question

I have searched all over, and read this post.

But it doesn't seem complete and doesn't work.

The situation: I need to get the last modified file from a directory on the local machine. I then need to pass that file into the fileinputdelimited component.

I currently have:

tfilelist --> iterate --> titeratetoflow --> tsamplerow -->tflowtoiterate -> tinpufiledelimited ---> tlogrow (just to make sure its pulling the right file)

But it doesn't work. I have configured it. so that titeratetoflow has a column called

• "FileName" with "((String)globalMap.get("CURRENT_FILE"))" as the value,
• "FileDirectory" with ((String)globalMap.get("CURRENT_FILEDIRECTORY")) as value, and
• "FileAndDirectory" with ((String)globalMap.get("CURRENT_FILEPATH")) as value.

The tsamplerow is limited to "1".

The tiflowtoiterate is set so that

• "FileNameOnly" is value of "FileName"
• "FileDirectoryOnly" is "FileDirectory" and
• "FilePathComplete" is "FileAndDirectory"

In the File location field of the tinputfiledelimited, I have "((String)globalMap.get("FilePathComplete"))"

When it runs I get an error saying cannot find file or path. If I cut out the fileinput component and have it send straight to the tlogrow, it shows a single line of blank entry.

Any ideas?

Answer 1

I'm not sure if you've just slightly misconfigured the job here but it seems to work fine for me.

Here's a few screenshots showing my job design:

The only thing I can think of just by looking at your post is that you might have slightly messed up the key value pair combinations in the tFlowToIterate. I tend to find that the default settings there work fine pretty much all of the time and it makes it a little more obvious what it's doing as well.

EDIT: Actually, it looks like you might be using the wrong values in your tIterateToFlow. The tFileList will throw the values for the file paths etc in to the global map but it will preface it with the unique component name. If you hit ctrl+space in the value window it should prompt you with a list of available values (these are also specified in the "Outline" tab of the studio). It typically makes an implicit conversion to String but for this you will need to explicitly convert it so use .toString() instead of (String).

Answer 2

Another way to get last modified file is as below

tFileList(sorted DESC by file modified date) ------> tFixedFlowInput (schema - filename, filenumber) ----->tHashOutput

here in tFixedFlowInput filename = file(String)globalMap.get("tFileList_1_CURRENT_FILEPATH")+"/"+(String)globalMap.get("tFileList_1_CURRENT_FILE")

filenumber = (Integer)globalMap.get("tFileList_1_NB_FILE")

What above will accomplish is get list of all files in the directory with their number/rank - where the file last modified will have file number =1 and next to that will have 2...and so on.

Now on SubJobOK of above tFileList you can have tHashInput which will read from above tHashOutput and filter only row where filenumber==1 - which means the last modified file.

tHashInput (link to tHashoutput) ---->tFilterRow(filenumber==1)------>tLogRow

One reason why you are getting null is probably you have used globalMap.get("CURRENT_FILEPATH) instead of globalMap.get("tFileList_1_CURRENT_FILEPATH")

Answer 3

The Simple Solution for above problem could be as below:

tFileList(sorted ASC by file modified date)--> tIterateToFlow --> tJava( just to end the subjob).

Then on

subjob ok --> tfileinput ( use (String)globalMap.get("tFileList_1_CURRENT_FILE") or (String)globalMap.get("tFileList_1_CURRENT_FILEPATH") as a file name/file path)

Explanation:

Since tFileList iterates all the files in ASC order, it will always have Latest file name stored in globalMap for the last iteration. The list is only iterated till tIterateToFlow hence after this component (String)globalMap.get("tFileList_1_CURRENT_FILE") will always give the last file name from the iterated list, which is the latest file in out case.

Main Flow :

Component View: