How to parametrize a transformation using a javascript calculated value?

Question

My real problem is a little more complicated, but what I'm trying to do it's supposed to be very basic so I tried to simplify everything as much as I can.

The problem

I need to calculate a value using a JavaScript function, and then I need to perform some operations on the database using this calculated value, for example my tranformation could look like this:

Modified Java Script Value

I have created a JavaScript function that calculates my value:

//Script here    
var calculated_value = "Hello, this is some value!";

this function returns one field, named calculated_value, of type String, and if I test it it looks fine:

Execute SQL script

I have then created a SQL script, using a parameter, something like:

INSERT INTO my_table (column) VALUES (?)

and I've set the calculated_value returned from the previous step as the first parameter

The problem

However, this is not working, if I launch my transformation I get an error stating that the parameter has not been set:

2014/05/03 15:44:26 - An error occurred, processing will be stopped:
2014/05/03 15:44:26 - Couldn't execute SQL: INSERT INTO my_table (valore) VALUES (?)
2014/05/03 15:44:26 -   
2014/05/03 15:44:26 - Statement parameter 1 not set.
2014/05/03 15:44:26 - Error initializing step [Execute SQL script]

Using variables?

I then decided to try again, using kettle variables. The JavaScript code just becomes:

//Script here
var calculated_value = "Hello, this is some value!";
setVariable("my_variable", calculated_value, "s");

and I've then modified the SQL script to use variable substitution:

INSERT INTO my_table (column) VALUES (${my_variable})

however, I still have no luck, the variable doesn't get substutited... or sometimes it gets substituted but with an old value and not the current one.

What am I doing wrong? How can I solve my problem?

I'm using Kettle - Spoon 4.4.0, but I'm planning to upgrade soon.

Answer 1

You are almost there. In order to use parameters with SQL step, you need to check "Execute for each row". This way the step will fire as many INSERT statement as stream input rows.

On a side note, I assume the logic behind your solution is deeper than the one coming out in the question, because in order to insert a value in a table there are more suitable steps (table output, insert/update).

Answer 2

This is a solution I found that makes use of variables.

It seems that variables cannot be set and used on the same transformation, so we need to create a job that calls a transformation that sets variables, and from this point all other transformations can make use of the variables that have been set:

Set Variable Transformation

The Java Script step from my question does not generate any row, it just generates one more column on an existing set of rows, so it can't be the first step of a transformation, but we have to add one empty row with a "Generate Row" step:

my JavaScript code is still the same:

//Script here    
var calculated_value = "Hello, this is some value!";

and to set my variable, I decidet to use the "Set Variables" step:

Execute SQL transformation

All transformations following the "Set Variable" transformation can now make use of variable substitution.

My SQL Script executes the query:

INSERT INTO my_table (column) VALUES ('${CALCULATED_VALUE}')

(please notice that I had to quote the variable, this could lead to SQL injections problems... but in my example it could be fine). I just had to set the "Variable substitution" flag and it works!

However, I'm still trying to make it work with parameters instead of variables.

Some reference can be found here: Using Variables in Kettle.