Data Pipeline script with while loop stuck in WAITING_FOR_DEPENDENCIES state

Question

I am trying to run the following script on an EC2 instance running Amazon Linux 64bit with a few extra things installed (lftp, for example) as part of a Amazon Data Pipeline in a ShellCommandActivity.

INPUT_FILE_LIST=`/bin/ls -1 ${INPUT1_STAGING_DIR}` && 
SFTP_FILE_LIST=`/usr/bin/lftp -u username,password -e "set cmd:cls-default -1 && cls && bye" sftp://sftp.server.com` && 
while read name; do if `/bin/echo "$INPUT_FILE_LIST" | /bin/grep -q "^$name$"`; then OLD_FILES="$OLD_FILES $name"; fi; done < <(/bin/echo "$SFTP_FILE_LIST") &&
if [[ $OLD_FILES ]]; then /usr/bin/lftp -u username,password -e "rm ${OLD_FILES} && bye" sftp://sftp.server.com; fi

If I remove the line with the while it will run to completion (the ShellCommandActivity reaches FINISHED status) but if it's there the script "fails" in the sense that the ShellCommandActivity ends up in a WAITING_ON_DEPENDENCIES status.

Unfortunately the Data Pipeline service is not writing out any logs in this case, so I'm not sure why I'm having problems, and I am able to run the command successfully if I create an instance with the same image and instance type and run the command myself by logging in to the box.

As is perhaps already obvious from the code, the goal of all this is to remove files in an S3 bucket from an sftp server.

Notes:

• INPUT1_STAGING_DIR is an S3 bucket and that part is managed by the Data Pipeline and I've already confirmed that this part is working
• The script is actually all executed on a single line; the lines are broken up to make it easier to run but when deployed all 4 lines get concatenated with just a space between each, hence the && at the end of each line and all the ; in the 3rd line.

Here is the code with nicer formatting for convenience:

INPUT_FILE_LIST=`/bin/ls -1 ${INPUT1_STAGING_DIR}` && 
SFTP_FILE_LIST=`/usr/bin/lftp -u username,password -e "set cmd:cls-default -1 && cls && bye" sftp://sftp.server.com` && 
while read name; do
  if `/bin/echo "$INPUT_FILE_LIST" | /bin/grep -q "^$name$"`; then
    OLD_FILES="$OLD_FILES $name";
  fi;
done < <(/bin/echo "$SFTP_FILE_LIST") &&
if [[ $OLD_FILES ]]; then
  /usr/bin/lftp -u username,password -e "rm ${OLD_FILES} && bye" sftp://sftp.server.com;
fi

Answer 1

I was able to work around this problem by using s3cmd in my script instead of an S3 resource. I'm not sure why this worked but it did. I just changed the first line to:

INPUT_FILE_LIST=`/usr/bin/s3cmd ls s3://my-bucket/my-prefix`

and changed the grep pattern in the third line to $name$ and it started working. Note that this seems odd since the script completes as written above if I remove the while loop.

Either way, problem "solved", although I'd be happy to replace this solution with one that is able to use the S3 data pipeline resource.

Answer 2

this works for me

input=$(aws s3 ls $1 | awk '{print  $4}')
echo "$input"