Pentaho table output step not showing proper error in log

Question

In Pentaho, I have a table output step where I load a huge num of records into a netezza target table.

One of the rows fails and the log shows me which values are causing the problem. But the log is probably not right, because when i create an insert statement with those values and run it separately on teh database, it works fine.

My question is: In Pentaho, is there a way to identify that when a db insert fails, exactly which values caused the problem and why?

EDIT: The error is 'Column width exceeded' and it shows me the values that is supposedly causing the problem. But I made an insert statement with those values and it works good. So I think Pentaho is not showing me the correct error message, it is a different set of values that are causing the problem.

Answer 1

Another way I've used to deal with these kind of problems is to create another table in the DB with widened column types. Then in your transform, add a Table output step connected to the new table. Then connect your original Table output to the new step, but when asked, choose 'Error handling' as the hop type.

When you run your transform, the offending rows will end up in the new table. Then you can investigate exactly what the problem is with that particular row.

For example you can do something like:

insert into [original table] select * from [error table];

You'll probably get a better error message from your native DB interface than from the JDBC driver.

Answer 2

I don't know what is your problem exactly, but I think I had the same problem before.

Everything seems right, but the problem was that in some tranformations, when I transform a numeric value to string for example, the transformation added a whitespace at the end of the field, and the long of the field was n+1 instead of n, but that is very difficult to see.

A practical example would be if you are transforming with a calculator step, you may use YEAR() function to extract the year of a date field, and maybe to that new field with the year have been added a whitespace, so if the year had a length of 4, after that step it will has a length of 5, and when you are going to load a row (with that year field that is a string(5)) into the data warehouse and in your data warehouse is expecting a string(4), you will get the same error that are getting now.

You think is happening --> year = "2013"   --> length 4
Really is happening    --> year = "2013 "  --> length 5

I recommend you to pay quite attention to the string fields and their lengths, because if some transformation adds a whitespace that you don't expect you can lose a lot of time to find the error (myself experience).

I hope this can be useful for you!

EDIT: I'm guessing you are working with PDI (Spoon, before Kettle) and the error is producing when you are loading a data warehouse, so correct me if I'm wrong.

Answer 3

Can you use the file with nzload command, with this command you can find exact error, and bad records in badFile provided by you for detailed analysis.

e.g. -

nzload -u <username> -pw <password> -host <netezzahost> -db <database> -t <tablename> -df <datafile> -lf <logfile> -bf <badrecords file name> -delim <delimiter>