Question

I have the following 3 commands (each contain xargs and parallel processes) that I would like to run collectively in parallel.

1. ls filenames_for_bld*_all | xargs -i bash -c 'calculateMd5Sums "$@"' _ {}
2. cat filenames_for_bld*_all | awk -F'|' '{print $2}' | xargs -n 1 -P 10 -I {} remotecmd -q -n server1 md5sum {} 2>/dev/null| tee -a ${dir}/md5sums_for_server1
3. cat filenames_for_bld*_all | awk -F'|' '{print $2}' | xargs -n 1 -P 23 -I {} remotecmd -q -n server2 md5sum {} 2>/dev/null | tee -a ${dir}/md5sums_for_server2

Each xargs command runs several processes on separate servers. So 1 runs on servers 1-5, 2 runs on server 6, and 3 runs on server 7.

Is it possible to run all of the above 3 in parallel?

Using xargs on the above would be nice. I was thinking of setting the above 3 to a string xargsInput, separated by newlines, then...

echo $xargsInput | xargs -n 1 -P 3 -I {} sh -c {}

However, this has some errors:

ls: xargs: No such file or directory
ls: bash: No such file or directory
ls: calculateMd5Sums: No such file or directory
ls: 20140107: No such file or directory
ls: _: No such file or directory
ls: {}: No such file or directory
Was it helpful?

Solution

Just add a & after each line, and optionally a wait at the end to wait for them all to finish:

ls filenames_for_bld*_all | xargs -i bash -c 'calculateMd5Sums "$@"' _ {} &
cat filenames_for_bld*_all | awk -F'|' '{print $2}' | xargs -n 1 -P 10 -I {} remotecmd -q -n server1 md5sum {} 2>/dev/null| tee -a ${dir}/md5sums_for_server1 &
cat filenames_for_bld*_all | awk -F'|' '{print $2}' | xargs -n 1 -P 23 -I {} remotecmd -q -n server2 md5sum {} 2>/dev/null | tee -a ${dir}/md5sums_for_server2 &
wait

This works for all separate shell commands that don't need to interact.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top