Wrong order of SQL statements on InnoDB from Kettle

Question

In Kettle, I use the following logic in a transformation, given some Strings X and Y as input:

[User Defined Java Expression] Generate ID
[Insert / Update] Update/Insert table set id = generatedId, name=X, company=Y where name = X; don't update the ID column
[Database Value Lookup]select id from table where name = X

Idea is to update existing entries in the table or create new ones and get the ID of the interesting row in the next step (which may be an existing one or the newly generated one).

This works fine when executed on MySQL + MyISAM but fails on MySQL + InnoDB, with all other parameters being identical. The last step fails when the row is just being inserted in the second step but works for rows already existing in the database. It seems as if the connection tries to execute the SELECT of the last step before the actual insert happened. All parameters are set to default in the MySQL settings (MySQL 5.1 and 5.5 show the same behavior).

So my questions are: What are the relevant parameters in Kettle and/or MySQL? How can I guarantee that this works as expected? I cannot switch back to MyISAM.

Answer 1

just use the block rows step between the insert step and the next step. Then the step before the block will complete before the next step starts.

Answer 2

Well, after having evaluated different possibilities, three seem to be possible:

1. Write my own step which performs the select/insert in a transaction
2. Serialize the whole transformation in its properties (makes everything REALLY slow)
3. Use Codeks idea and use the blocking step

I went with the third option for now as everything else is not possible for the moment.

Answer 3

Make sure the transaction generated by Update/Insert is committed and the locks are released before doing the SELECT operation takes place. It looks like there are lock problems