SFTP - capture file list not working as expected

Question

We want to capture a file list (whether 1 or more) in BASH, evaluate if the file has changed..

1. If the file has changed/updated
2. Delete the previously downloaded file
3. Download the new file
4. Notify a group of people (to do)

I have seen this.. save the result of ls command in the remote sftp server on local machine

but, seems to not work correctly for what we want to do..

Environment

LINUX: RedHat (2.6.xx-xxx.x.x.xxx)

SFTP:

• SFTP(1) BSD General Commands Manual
• SFTP protocol version 3

Using the link from above:

sftp account@0.0.0.0
expect "password:"
send "somepass\n";
set file [open /home/checks/sftp_files w]
send  "ls /home/sftp_server_location/*.zip\n"
log_file
interact

I have been reading that folks have had issue with SFTP creating a list of files.. FTP - come to find out, does not have this issue.

We are doing this already with WinSCP under Windows and works fine.

However, the systems and datacenter this is in, is being shutdown and we need to get this on a LINUX box..

With Windows, we are doing on a high-level:

Windows Scheduled Task that runs daily
To create a list of files beforehand: dir "d:\Downloads\*.gz" /B/S  > D:\scripts\OLDLIST.TXT
Pause for 1 second: PING 127.0.0.1 -w 1000 > NUL
Run a script for WinSCP: cscript /nologo D:\scripts\somscript.js
Pause for 1 second: PING 127.0.0.1 -w 1000 > NUL
Create a list of files after: dir "d:\GEO_IP_Downloads\*.gz" /B/S  > D:\scripts\NEWLIST.TXT
Pause for 1 second: PING 127.0.0.1 -w 1000 > NUL
Run a FC (File Compare):
%windir%\system32\FC /B D:\scripts\NEWLIST.TXT D:\scripts\OLDLIST.TXT | %windir%\system32\FIND /i "FC: D:\SCRIPTS\NEWLIST.TXT longer than D:\SCRIPTS\OLDLIST.TXT"
If there is a new file found, then e-mail:
IF NOT ERRORLEVEL 1 d:\SCRIPTS\BLAT.exe -body "New file(s) have been found at Vendor" -f somebody@fisglobal.com -to sombodyelse@fisglobal.com -server 1.2.3.4 -subject "New file(s) have been found at company" -q

WinSCP is from – https://winscp.net

The specific script we are using is from –
https://winscp.net/eng/docs/script_download_most_recent_file#scripting

Answer 1

Here's a rough version of what you're looking for. It will require you to keep a list from the last sync around. (How you authenticate for this is a different topic, but instead of saving the password in a script I'd recommend using ssh keys).

#!/bin/bash

lastlist=/tmp/sftp_list.old
newlist=/tmp/sftp_list.new
localpath=/home/foo/downloads
remotepattern='/home/bar/*.zip'
remotehost='account@0.0.0.0'
notifysubject="New remote files"
notifyaddress="foo@example.com bar@example.com"

cd "$localpath"
if echo "ls -1 $remotepattern" | sftp "$remotehost" >"$newlist"; then
    filelist=$(comm -13 "$lastlist" "$newlist")
    [ $(echo "$filelist" | wc -l) -gt 0 ] && {
        {
            echo "New remote files:"
            echo "$filelist"
        } | mail -s "$notifysubject" "$notifyaddress"

        # download new files
        echo "$filelist" | sed 's/^/get /' | sftp "$remotehost"

        rm -f "$lastlist"
        mv "$newlist" "$lastlist"
    }
else
    echo "Error connecting to the remote host"
fi

This downloads only new files, but it will fail in subtle ways if any of the filenames contain \n (which is very rare). A simple fix is to re-download all matching files instead of selecting by name: echo "get $remotepattern" | sftp "$remotehost"