Running same Kettle Job from two different scripts Issue

Question

Is it possible to run a kettle job simultaneously more than once at the same time?

What I am Trying

Say we are running this script twice at a same time,

sh kitchen.sh -rep="development" -dir="job_directory" -job="job1"

If I run it only once at a point of time, data-flow is perfectly fine.

But, when I run this command twice at the same time, it throws error like:

ERROR 09-01 13:34:13,295 - job1 - Error in step, asking everyone to stop because of:
ERROR 09-01 13:34:13,295 - job1 - org.pentaho.di.core.exception.KettleException: 
java.lang.Exception: Return code 1 received from statement : mkfifo /tmp/fiforeg
Return code 1 received from statement : mkfifo /tmp/fiforeg

    at org.pentaho.di.trans.steps.mysqlbulkloader.MySQLBulkLoader.execute(MySQLBulkLoader.java:140)
    at org.pentaho.di.trans.steps.mysqlbulkloader.MySQLBulkLoader.processRow(MySQLBulkLoader.java:267)
    at org.pentaho.di.trans.step.RunThread.run(RunThread.java:50)
    at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.Exception: Return code 1 received from statement : mkfifo /tmp/fiforeg
    at org.pentaho.di.trans.steps.mysqlbulkloader.MySQLBulkLoader.execute(MySQLBulkLoader.java:95)
    ... 3 more

It's important to run the jobs simultaneously twice at a same time. To accomplish this, I can duplicate every job and run the original and the duplicate job at a point of time. But, not a good approach for long run!

Question:

1. Is Pentaho not maintaining threads?

2. Am I missing some option, or can I enable some option to make pentaho create different threads for different job instances?

Answer 1

Of course Kettle maintains threads. A great many of them in fact. It looks like the problem is that the MySQL bulk loader uses a FIFO. You have two instances of a FIFO called /tmp/fiforeg. The first instance to run creates the FIFO just fine; the second then tries to create another instance with the same name and that results in an error.

At the start of the job, you need to generate a unique FIFO name for that instance. I think you can do this by adding a transformation at the start of the job that uses a Generate random value step to generate a random string or even a UUID and store it in a variable in the job via the Set variables step.

Then you can use this variable in the 'Fifo file' field of the MySQL bulk loader.

Hope that works for you. I don't use MySQL, so I have no way to make sure.