Python Paramiko Callback - duplicate output at 100% during file transfer

Question

I am using paramiko to access a network device and retrieve files. This is working ok. I added the callback function, to get progress on each file xfer. And I see that for each file that is transferred, it seems to report twice that it has finished the transfer.

Here is the code.

from __future__ import division
import paramiko, os

def byte_count(xfer, to_be_xfer):
    print net_file + " transferred: {0:.0f} %".format((xfer / to_be_xfer) * 100)

path = '/var/log/VPN*'
cmd = 'file list ' + path

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('10.10.10.10', username = uname, password = psswd)
stdin, stdout, stderr = ssh.exec_command(cmd)

filelist = stdout.read().splitlines()

ftp = ssh.open_sftp()
for f in filelist:
    net_path,net_file=os.path.split(f)
    ftp.get(f, 'c:\\temp\\' + net_file, callback=byte_count) 
ftp.close()

print '\nExiting..'

ssh.close()

I get this as a result...

VPN transferred: 14 %
VPN transferred: 28 %
VPN transferred: 42 %
VPN transferred: 56 %
VPN transferred: 69 %
VPN transferred: 83 %
VPN transferred: 97 %
VPN transferred: 100 %
VPN transferred: 100 %
VPN.0.gz transferred: 100 %
VPN.0.gz transferred: 100 %
VPN.1.gz transferred: 44 %
VPN.1.gz transferred: 88 %
VPN.1.gz transferred: 100 %
VPN.1.gz transferred: 100 %
VPN.2.gz transferred: 43 %
VPN.2.gz transferred: 87 %
VPN.2.gz transferred: 100 %
VPN.2.gz transferred: 100 %

Why is the last line reported twice? Is there anyway to fix this?

Answer 1

This seems to be a bug in Paramiko. See the source code for paramiko v1.12.0 and how the callback is called before checking if the transfer is already finished:

def getfo(self, remotepath, fl, callback=None):
    fr = self.file(remotepath, 'rb')
    file_size = self.stat(remotepath).st_size
    fr.prefetch()
    try:
        size = 0
        while True:
            data = fr.read(32768)
            fl.write(data)
            size += len(data)
            if callback is not None:
                callback(size, file_size)
            if len(data) == 0:
                break
    finally:
        fr.close()
    return size

In later versions this is fixed - http://www.lag.net/paramiko/docs/paramiko.sftp_client-pysrc.html

def get(self, remotepath, localpath, callback=None): 
  fr = self.file(remotepath, 'rb') 
  file_size = self.stat(remotepath).st_size 
  fr.prefetch() 
  try: 
      fl = file(localpath, 'wb') 
      try: 
          size = 0 
          while True: 
              data = fr.read(32768) 
              if len(data) == 0: 
                  break 
              fl.write(data) 
              size += len(data) 
              if callback is not None: 
                  callback(size, file_size) 
      finally: 
          fl.close() 
  finally: 
      fr.close() 

See the difference -

if len(data) == 0:
    break

is called before the callback in the latter. So you have to either get a newer version of Paramiko, or to live with the bug... Or maybe to use a patch for Paramiko code :)