How to submit a cat command to a cluster using qsub and use the pipe correctly

Question

I want to submit several "cat jobs" on the fly to a cluster using qsub. Currently I'm concatenating several files with cat to a single one at the end (using > output_file) at the end of the command. The problem is that qsub takes the > output_file from the command as part of the qsub, putting there the log of the job instead of the cat output.

qsub -b y -cwd -q bigmem cmd1

where cmd1 looks like:

cat file1 file2 filen > output_file

Answer 1

When a job is running via pbs, stdout is redirected to the output file of the job, so the way to do this is to write a script:

#!/bin/bash
cat file1 file2 ... filen

You don't need to redirect the output to a file because the mom daemon will do that for you in setting up the job, you just need to specify the output file you desire using -o. For example, if you named the above script script.sh (make sure it is executable) you'd submit:

qsub script.sh -b y -q bigmem -o output_file

Answer 2

Alternatively to dbeer's answer, if your code is disposable, you can use echo:

echo "cat file1 file2 ... filen > outfile" | qsub -cwd <options>