scp: how to find out that copying was finished

Question 1

Off the top of my head, you could do something like:

touch tinyfile
scp bigfile tinyfile user@host:

Then when tinyfile appears you know that the transfer of bigfile is complete.

As pointed out in the comments, this assumes that scp will copy the files one by one, in the order specified. If you don't trust it, you could do them one by one explicitly:

scp bigfile user@host:
scp tinyfile user@host:

The disadvantage of this approach is that you would potentially have to authenticate twice. If this were an issue you could use something like ssh-agent.

Question 2

On sending side (host1) use script like this:

#!/bin/bash

echo 'starting transfer'
scp FILE USER@DST_SERVER:DST_PATH
OUT=$?
if [ $OUT = 0 ]; then
  echo 'transfer successful'
  touch successful
  scp successful USER@DST_SERVER:DST_PATH
else
  echo 'transfer faild'
fi

On receiving side (host2) make script like this:

#!/bin/bash 

SLEEP_TIME=30
MAX_CNT=10
CNT=0
while [[ ! -e successful && $CNT < $MAX_CNT ]]; do
    ((CNT++))

    sleep($SLEEP_TIME);
done; 

if [[ -e successful ]]; then
    echo 'successful'
    rm successful

    # do somethning with FILE
fi

With CNT and MAX_CNT you disable endless loop (in case file successful isn't transferred). Product MAX_CNT and SLEEP_TIME should be equal or greater expected transfer time. In my example expected transfer time is less than 300 seconds.

Question 3

A checksum (md5sum, sha256sum ,sha512sum) of the local and remote files would tell you if they're identical.

For the situation where you don't have SSH access to the remote system - like an FTP server - you can download the file after it's uploaded and compare the checksums. I do this for files I send from production scripts at work. Below is a snippet from the script in which I do this.

MD5SRC=$(md5sum $LOCALFILE | cut -c 1-32)
MD5TESTFILE=$(mktemp -p /ramdisk)
curl \
    -o $MD5TESTFILE \
    -sS \
    -u $FTPUSER:$FTPPASS \
    ftp://$FTPHOST/$REMOTEFILE
MD5DST=$(md5sum $MD5TESTFILE | cut -c 1-32)
if [ "$MD5SRC" == "$MD5DST" ]
then
    echo "+Local and Remote files match!"
else 
    echo "-Local and Remote files don't match"
fi

Question 4

if you use inotify-tools, then the solution will looks like this:

while ! inotifywait -e close $(dirname ${bigfile_fullname}) 2>/dev/null | \
    grep -Eo "CLOSE $(basename ${bigfile_fullname})$">/dev/null
do true
done
echo "File ${bigfile_fullname} closed"

Question 5

After some investigation, and discussion of the problem on other forums I have found one more solution. Maybe it can help somebody.

There is a command "lsof". It lists open files. During copying the file will be opened, so the command

lsof | grep filename

will return non empty result.

So you might want to make a while loop to wait until lsof returns nothing and proceed with your task.

Example:

# provide your file name here
f=<nameOfYourFile>
lsofresult=`lsof | grep $f | wc -l`
while [ $lsofresult != 0 ]; do 
  echo still copying file $f...
  sleep 5
  lsofresult=`lsof | grep $f | wc -l`
done; echo copying file $f is finished: `ls $f`

Question 6

For the duplicate question, How to check if file has been scp 100% to the remote location , which was for an expect script, to know if a file is transferred completely, we can add expect 100% .. .. i.e something like this ...

expect -c "  
   set timeout 1
   spawn scp user@$REMOTE_IP:/tmp/my.file user@$HOST_IP:/home/.
   expect yes/no { send yes\r ; exp_continue }
   expect password: { send $SCP_PASSWORD\r }
   expect 100%
   sleep 1
   exit
"  
if [ -f "/home/my.file" ]; then  
echo "Success"  
fi

Question 7

If avoiding a second SSH handshake is important, you can use something like the following:

ssh host cat \> bigfile \&\& touch complete < bigfile

Then wait for the "complete" file to get created on the remote end.

Question 8

just use rsync instead--as it will write to a hidden temporary file, then rename that file when the copy is complete