Are paramiko file transfer operations atomic?

Question

I use get and put methods of paramiko.SFTP object. I can vaguely remember that I did at times get incomplete file contents with the get method, but never noticed any issue with put. Is it just my luck, or should I perform extra checks to ensure that I get the right files on the remote side? If so, how do I do it, given I only have SFTP access to that remote server?

By SFTP-only access, I mean I cannot SSH in:

$ ssh user@ipaddress
user@ipaddress's password: 
PTY allocation request failed on channel 0
shell request failed on channel 0

Answer 1

You can be fairly certain if an SFTP transfer completed without error or interruption, that the file has been fully copied over, since the SFTP protocol requires that each chunk sent be acknowledged. So, between the reliability of TCP, the SSH channel, and the ACKs from each chunk, it's very unlikely that the file has been truncated in any way.

A couple more things you could do when only using SFTP:

• After transfer, you can call stat() on the file to verify it's actual size.
• You could read back one or more chunks of the file and compare them locally.
• Some SFTP servers have a check-file extension (I don't think openssh supports this though). Paramiko sftp files have a check() method that allows you to get a hash of some or all of the file.

And to directly answer your question's title, transfers are not atomic. The file could still be altered on the system as it's being written. If you want to prevent against that, you could write the file to a temporary location remotely, verify it, then call sftp.rename to move it to it's final destination (which is an atomic operation on Linux within the same filesystem). That's roughly analogous to what rsync does, and probably as close as you can come to "atomic".