use GNU parallel in a pipe

Question

Running the following command does what I want when reading from a file:

parallel --gnu -j2 "echo {} && sleep 5" < myfile.txt

I would like to do something similar with a pipe. Note that I used the following page for inspiration for the pipe reader and writer: http://www.linuxjournal.com/content/using-named-pipes-fifos-bash

Here is my pipe reader file:

#!/bin/bash

pipe=/tmp/testpipe

trap "rm -f $pipe" EXIT

if [[ ! -p $pipe ]]; then
    mkfifo $pipe
fi

while true
do
    parallel --gnu -j2 "echo {} && sleep 5" <$pipe
done

echo "Reader exiting"

And here is my writer:

#!/bin/bash

pipe=/tmp/testpipe

if [[ ! -p $pipe ]]; then
    echo "Reader not running"
    exit 1
fi


if [[ "$1" ]]; then
    echo "$1" >$pipe
else
    echo "Hello from $$" >$pipe
fi

I then run pipeWriter several times, such as

$ ./pipeWriter one
$ ./pipeWriter two
$ ./pipeWriter three
$ ./pipeWriter four
$ ./pipeWriter five
$ ./pipeWriter six
$ ./pipeWriter seven
$ ./pipeWriter eight
$ ./pipeWriter nine
$ ./pipeWriter ten
$ ./pipeWriter ww
$ ./pipeWriter wwdfsdf
$ ./pipeWriter wwdfsdfsddfsd
$ ./pipeWriter testAgain

The shell running pipeReader shows:

$ ./pipeReader 
one
four
eight
ww
testAgain

First, there is a problem of missing data. Second, parallel does not seem to run in parallel when in the pipe. I would like it to run two jobs at a time (or rather, at most two jobs. If it only has one job that's fine and if another it can start it).

Where am I going wrong?

Answer 1

I cannot explain the missing data, and when I run it I do not get missing data. I can, however, explain why you do not see the jobs run in parallel: Your while-loop in the reader may pass only one argument to parallel.

Instead of a pipe use a file as described in http://www.gnu.org/software/parallel/man.html#example__gnu_parallel_as_queue_system_batch_manager

Maybe you can also use cat:

while true
do
    cat $pipe
done | parallel --gnu -j2 "echo {} && sleep 5"

If you have not already done so, walk through the tutorial; your command line will love you for it: http://www.gnu.org/software/parallel/parallel_tutorial.html