sun grid engine synchronization point

Question

I am creating a set of tasks in a cluster using the -qsub command.

Inside my matlab code I would like to make a synchronization point to check whether all of my workers are finished execution or not. If all have finished execution I want to assign some other tasks to them.

For e.g.:my function (matlab) is:

        function test(taskId)
            do_task_1(taskId);
            __sync() ----->check whether all the workers have finished the job successfully
            do_task_2(taskId);
        end

How can I do it? p.s. I am a beginner to cluster computing.

Answer 1

A little late to the game, but might still be useful to you or others. I doubt you can do this from within matlab. Others with more experience may suggest otherwise. I would split the two tasks into separate matlab programs and use the -sync option with Grid Engine:

qsub -sync y do_task_1.sh
# -sync y makes qsub wait for job to finish
qsub do_task_2.sh

You can put the above commands in a script so you don't have to wait for first task to complete.

Alternatively, you can use '-hold_jobid`:

qsub do_task_1.sh
qsub -hold_jid <job_id_of_task_1> do_task_2.sh