How to remote ls on list of files from bash shell script?

Question

I have a below shell script from which I am trying to copy 5 files in parallel. I am running my below shell script on machineA which tries to copy the file from machineB and machineC.

If the file is not there in machineB, then it should be there in machineC for sure.

I am using GNU Parallel here to download five files in parallel. And everything works fine if all the files are there -

#!/bin/bash

export PRIMARY=/data01/primary
export FILERS_LOCATION_1=machineB
export FILERS_LOCATION_2=machineC
export MEMORY_MAPPED_LOCATION=/bexbat/data/be_t1_snapshot
PRIMARY_PARTITION=(550 274 2 546 278 6 558 282 10 554 286 14) # this will have more file numbers

export dir1=/bexbat/data/re_t1_snapshot/20140501

# just iterating the file and doing ls and exit if any of the file is missing
for el in "${PRIMARY_PARTITION[@]}"
do
    ssh david@$FILERS_LOCATION_1 ls $dir3/t1_weekly_1680_"$el"_200003_5.data || ssh david@$FILERS_LOCATION_2 ls $dir3/t1_weekly_1680_"$el"_200003_5.data || echo "File number $el missing on both the filers for primary partition." >&2; exit 1
done

echo "All files present. Starting to copy now."

# copy the files now

Problem Statement:-

Before copying any files, I want to see whether all the files are already present in either of the machines (machineB or machineC) or not. If any of the file is missing, then I need to print out which file is missing and exit out of the shell script with non zero status.

Above script is not working as the way I have described. If it sees any of the file is present, then it exits automatically, it's not moving in the for loop to look for other files. And I am not sure why?

Is there anything wrong I am doing?

Answer 1

ssh doesn't preserve quoting, so you need to escape commands locally to be unescaped by the remote shell.

for el in "${PRIMARY_PARTITION[@]}"
do
    printf -v cmd '%q ' test -e "$dir3/t1_weekly_1680_${el}_200003_5.data"
    ssh "david@$FILERS_LOCATION_1" "$cmd" \
    || ssh "david@$FILERS_LOCATION_2" "$cmd" \
    || { echo "File number $el missing on both the filers for primary partition." >&2;
         exit 1; }
done

Answer 2

That ssh line in the loop doesn't do what you expect. The semi-colon has a lower precedence than the other operators, so when you suffix the line with ; exit 1 that will be executed always. You could just use an if statement:

if ! ssh david@$FILERS_LOCATION_1 ls $dir3/t1_weekly_1680_"$el"_200003_5.data && \
   ! ssh david@$FILERS_LOCATION_2 ls $dir3/t1_weekly_1680_"$el"_200003_5.data;
then
    echo "File number $el missing on both the filers for primary partition." >&2
    exit 1
fi