File I/O in gnu parallel

Question

I have a program that takes a single argument. I am using gnu parallel to perform parameter sweeps on this argument. Each run generates a single result, and I want to append all results into a single file, say Results.txt.

What would be a correct way to do this?

I should not have each instance open the file and write to it, as this could create conflicts and also mess up the order of results. The only way I can think of doing this is having each run generate its output in a file with a unique name, and then , when gnu parallel finishes running, merge the results into a single file using a script.

1. Is there a simpler way of achieving this?
2. What happens when multiple instances write to/read from the same file? Does gnu parallel create multiple copies, one for each instances, as it does for stdout and stderror?

thanks

Answer 1

If your command sends the result to stdout (standard output) the solution is trivial:

seq 1000 | parallel echo > Results.txt

GNU Parallel guarantees the output will not be mixed.

Answer 2

Normally GNU Parallel prints the output of a job as soon as it's completed. When jobs run for a different amount of time, this can lead to their output being mixed.

To keep the output in order, simply add -k / --keep-order parameter.

Try for example:

parallel -j4 sleep {}\; echo {} ::: 2 1 4 3
parallel -j4 -k sleep {}\; echo {} ::: 2 1 4 3