Directly pass parameters to pbs script

Question

Is there a way to directly pass parameters to a .pbs script before submitting a job? I need to loop over a list of files indicated by different numbers and apply a script to analyze each file.

The best I've been able to come up with is the following:

#!/bin/sh 

for ((i= 1; i<= 10; i++))
do
        export FILENUM=$i
        qsub pass_test.pbs
done

where pass_test.pbs is the following script:

#!/bin/sh

#PBS -V
#PBS -S /bin/sh
#PBS -N pass_test
#PBS -l nodes=1:ppn=1,walltime=00:02:00
#PBS -M XXXXXX@XXX.edu

cd /scratch/XXXXXX/pass_test

./run_test $FILENUM

But this feels a bit wonky. Particularly, I want to avoid having to create an environment variable to handle this.

Answer 1

The qsub utility can read the script from the standard input, so by using a here document you can create scripts on the fly, dynamically:

#!/bin/sh

for i in `seq 1 10`
do
    cat <<EOS | qsub -
#!/bin/sh

#PBS -V
#PBS -S /bin/sh
#PBS -N pass_test
#PBS -l nodes=1:ppn=1,walltime=00:02:00
#PBS -M XXXXXX@XXX.edu

cd /scratch/XXXXXX/pass_test

./run_test $i
EOS
done

Personally, I would use a more compact version:

#!/bin/sh

for i in `seq 1 10`
do
    cat <<EOS | qsub -V -S /bin/sh -N pass_test -l nodes=1:ppn=1,walltime=00:02:00 -M XXXXXX@XXX.edu -
cd /scratch/XXXXXX/pass_test
./run_test $i
EOS
done

Answer 2

You can use the -F option, as described here:

-F

Specifies the arguments that will be passed to the job script when the script is launched. The accepted syntax is:

qsub -F "myarg1 myarg2 myarg3=myarg3value" myscript2.sh

Note: Quotation marks are required. qsub will fail with an error message if the argument following -F is not a quoted value. The pbs_mom server will pass the quoted value as arguments to the job script when it launches the script.

See also this answer

Answer 3

If you just need to pass numbers and run a list of jobs with the same command except the input file number, it's better to use a job array instead of a for loop as job array would have less burden on the job scheduler.

To run, you specify the file number with PBS_ARRAYID like this in the pbs file:

./run_test ${PBS_ARRAYID}

And to invoke it, on command line, type:

qsub -t 1-10 pass_test.pbs

where you can specify what array id to use after -t option