Spring Batch: different job launcher for different jobs

Question

I have 2 different jobs (actually more but for simplicity assume 2). Each job can run in parallel with the other job, but each instance of the same job should be run sequentially (otherwise the instances will cannibalize eachother's resources).

Basically I want each of these jobs to have it's own queue of job instances. I figured I could do this using two different thread pooled job launchers (each with 1 thread) and associating a job launcher with each job.

Is there a way to do this that will be respected when launching jobs from the Spring Batch Admin web UI?

Answer 1

There is a way to specify a specific job launcher for a specific job, but the only way I have found to do it is through use of a JobStep.

If you have a job called "specificJob" this will create another job "queueSpecificJob" so when you launch it, either through Quartz or Spring Batch web admin, it will queue up a "specificJob" execution.

<bean id="specificJobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
    <property name="jobRepository" ref="jobRepository"/>
    <property name="taskExecutor">
        <task:executor id="singleThreadPoolExecutor" pool-size="1"/>
    </property>
</bean>

<job id="queueSpecificJob">
    <step id="specificJobStep">
        <job ref="specificJob" job-launcher="specificJobLauncher" job-parameters-extractor="parametersExtractor" />
    </step>
</job>

Answer 2

@ ahbutfore

How are the jobs triggered? Do you use Quartz trigger by any chance? If yes, would implementing/extending the org.quartz.StatefulJob interface in all your jobs do the work for you?

See Spring beans configuration here : https://github.com/regunathb/Trooper/blob/master/examples/example-batch/src/main/resources/external/shellTaskletsJob/spring-batch-config.xml. Check source code of org.trpr.platform.batch.impl.spring.job.BatchJob

You can do more complex serialization (including across Spring batch nodes) using a suitable "Leader Election" implementation. I have used Netflix Curator (an Apache Zookeeper recipe) in my project. Some pointers here : https://github.com/regunathb/Trooper/wiki/Useful-Batch-Libraries

Answer 3

Using a shell script you can launch different jobs parallel.

Add an '&' to the end of each command line. The shell will execute them in parallel with it's own execution.