Pentaho Data Integration: How to select output of sql query as a filename for Microsoft Excel Input.

Question

I have files abc.xlsx, 1234.xlsx, and xyz.xlsx in some folder. My requirement is to develop a transformation where the Microsoft Excel Input in PDI (Pentaho Data Integration) should only pick the file based on the output of a sql query. If the output query is abc.xlsx. Microsoft Excel Input should pick of abc.xlsx for further processing. How do I achieve this? Would really appreciate your help. Thanks.

Answer 1

Transformations in Kettle run asynchronously, so you're probably looking into needing a job for this.

Files to create

• Create a transformation that performs the SQL query you're looking for and populates a variable based on the result
• Create a transformation that pulls data from the Excel file, using the variable populated as the filename
• Create a job that executes the first transformation, then steps into the second transformation

Jobs run sequentially, so it will execute the first transformation, perform the query, get the result, and set a variable. Variables need to be set and retrieved in different transformations because of their asynchronous nature. This is the reason for the second transformation; the job won't step into the second transformation until the first one is done running (therefore, not until the variable is populated).

This is all assuming you only want to run the transformation once, expecting a single result from the query. If you want to loop it, pulling data from a set, then setup is a little bit different.

Answer 2

The Excel input step has a "accept filenames from previous step" option. You can have a table input build the full path of the file you want to read (or you somehow build it later knowing the base dir and the short filename), pass the filename to the excel input, tick that box and specify the step and the field you want to use for the filename.