bash - errors trying to pipe commands to run to separate function

Question

I'm trying to get this function for making it easy to parallelize my bash scripts working. The idea is simple; instead of running each command sequentially, I pipe the command I want to run to this function and it does while read line; run the jobs in the bg for me and take care of logistics.... it doesn't work though. I added set -x by where stuff's executed and it looks like I'm getting weird quotes around the stuff I want executed... what should I do?

runParallel () {
    while read line
    do
        while [ "`jobs | wc -l`" -eq 8 ]
        do
            sleep 2
        done

        {
            set -x
            ${line}
            set +x
        } &
    done

    while [ "`jobs | wc -l`" -gt 0 ] 
    do
        sleep 1
        jobs >/dev/null 2>/dev/null
        echo sleeping
    done
}

for H in `ypcat hosts | grep fmez | grep -v mgmt | cut -d\  -f2 | sort -u`
do
    echo 'ping -q -c3 $H 2>/dev/null 1>/dev/null && echo $H - UP || echo $H - DOWN'
done | runParallel

When I run it, I get output like the following:

> ./myscript.sh
+ ping -q -c3 '$H' '2>/dev/null' '1>/dev/null' '&&' echo '$H' - UP '||' echo '$H' - DOWN
Usage: ping [-LRUbdfnqrvVaA] [-c count] [-i interval] [-w deadline]
            [-p pattern] [-s packetsize] [-t ttl] [-I interface or address]
            [-M mtu discovery hint] [-S sndbuf]
            [ -T timestamp option ] [ -Q tos ] [hop1 ...] destination
+ set +x
sleeping
> 

Answer 1

The quotes in the set -x output are not the problem, at most they are another result of the problem. The main problem is that ${line} is not the same as eval ${line}.

When a variable is expanded, the resulting words are not treated as shell reserved constructs. And this is expected, it means that eg.

A="some text containing > ; && and other weird stuff"
echo $A

does not shout about invalid syntax but prints the variable value.

But in your function it means that all the words in ${line}, including 2>/dev/null and the like, are passed as arguments to ping, which set -x output nicely shows, and so ping complains.

If you want to execute from variables complicated commandlines with redirections and conditionals, you will have to use eval.

Answer 2

If I'm understanding this correctly, you probably don't want single quotes in your echo command. Single quotes are literal strings, and don't interpret your bash variable $H.

Answer 3

Like many users of GNU Parallel you seem to have written your own parallelizer.

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

cat hosts | parallel -j8 'ping -q -c3 {} 2>/dev/null 1>/dev/null && echo {} - UP || echo {} - DOWN'

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Answer 4

Put your command in an array.