GNU parallel output progress while output to file

Question

I have a simple bash script to run:

cat full_path.csv | parallel --progress -j +0 'echo -n {},; pdfgrep -c [^_] {};' > path_count.csv

Parallel's progress indicator "--progress", writes into the file path_count.csv. I only want echo {} and pdfgrep {} to write to the file, while showing --progress output to screen.

If I do :

cat full_path.csv | parallel --progress -j +0  'echo -n {},>>path_count.csv; pdfgrep -c [^_] {}>>path_count.csv;'

the file path_count is still garbled with progress.

Any help is appreciated. Thanks Alvin

Answer 1

The behaviour you see is not what GNU Parallel is designed to do: --progress is normally sent to STDERR and not to STDOUT for exactly that reason:

$ seq 3 | bin/parallel --progress echo {} >/tmp/out

Computers / CPU cores / Max jobs to run
1:local / 8 / 3

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
local:0/3/100%/0.0s
$ cat /tmp/out
1
2
3

Has there been local modifications of GNU Parallel? Can you reproduce the issue on other systems?

PS: instead of 'echo -n' why to try: --tag

Answer 2

Try to redirect it completely within the subshell using exec:

: > path_count.csv  ## truncate file
cat full_path.csv | parallel --progress -j +0 'exec >>path_count.csv; echo -n {},; pdfgrep -c [^_] {};'