pentaho column level validation

Question

I am new in Pentaho project. Need to know about some validation rules whether these can be handled by PDI or not. My input data is CSV format and target is ORACLE table. Suppose total number of column is 50.

Validation Rules :

• if any less number of columns coming from input file (like 48) we should inform via email
• if any more number of columns coming from input file (like 54) we should inform via email
• if there are any duplicate columns coming in input file we should inform via email
• if there a missing order in input file with respect to target table layout we should inform via email
• if there any columns started with a number we should inform via email
• if there any columns more than 30 characters in length we should inform via email

Any help is appreciated.

If anything required from me kindly reply.

Answer 1

Yes. There are steps that will allow you to identify all of these things.

More generally, if you want to do something in Kettle and don't see a step available that makes it easy to do, you can always write custom steps or steps with custom Java or JavaScript code in them to do it.

Answer 2

After lots of R&D it can be done by adding "Metadata Structure" under Utility in Transformation and load all column list into a text file.

Then create a Job and compare this text file with lookup text file (containing all target column list) by "File Compare" under File Management. Make a check option if true then invoke the final Transformation or if False send email notification.