How to determine the end of a file when using RETR on FTP server (PASV)

Question

I've decided to roll my own FTP Socket client in C# in order to have more control but mostly for the learning experience. But I cannot figure out how much file data I need to read after issuing the RETR command. I am able to establish the initial FTP communication connection, and then connect a second socket on a new port by issuing the PASV command and all that good stuff.

My problem is, when I issue the RETR command on the Communication port, the FTP server begins sending the file over the Passive port and then issues the 226 command when it finishes sending on the Communication port. I can receive the data fine but sometimes the 226 command arrives on the Communication port before all of the file data arrives on the Passive port. So, if I immediately signal my Passive socket task to stop reading after I get the 226 command, it will sometimes miss out on the data that comprises the end of the file. I've tried checking for Socket.Available before exiting out of my Socket.Receive loop but more often than not Socket.Available will report 0 bytes available on the socket but with sockets that doesn't mean there isn't more data on the way. Reading the FTP spec it seems as though the FTP server is supposed to shut down the data connection after it finishes sending all the data but I don't think that is happening. I can keep looping reading 0 bytes of data all day long. Am I looking at a broken FTP server or am I doing something wrong in my methodology?

Now for some additional info that might be the answer to my question. Using Filezilla, connected to the same FTP server, I see that when requesting a file the FTP server appends the file length to the 150 command in parenthesis. For example, "150 Opening BINARY mode data connection for test.zip (123421 bytes)." Is this text part of the FTP standard? If all FTP servers use this same behavior and formatting then I can just keep reading until I hit 123,421 bytes. But if this isn't common behavior then I'm back at square one. Appreciate any ideas.

Answer 1

I cannot figure out how much file data I need to read after issuing the RETR command.

It depends on the transfer mode (see RFC 959 Section 3.4).

In STREAM mode (the usual case), the end of file is denoted by the sender closing the data socket. This means the data connection cannot be reused for multiple transfers.

In BLOCK mode, the data is sent in blocks, and the sender will send an EOF block at the end of the transfer. This allows the data connection to be reused for multiple transfers.

My problem is, when I issue the RETR command on the Communication port, the FTP server begins sending the file over the Passive port and then issues the 226 command when it finishes sending on the Communication port. I can receive the data fine but sometimes the 226 command arrives on the Communication port before all of the file data arrives on the Passive port.

Simply don't read the response on the communication port until after you have finished reading the data on the transfer port.

So, if I immediately signal my Passive socket task to stop reading after I get the 226 command, it will sometimes miss out on the data that comprises the end of the file.

So don't signal your socket to stop reading. Let it stop reading on its own.

I've tried checking for Socket.Available before exiting out of my Socket.Receive loop but more often than not Socket.Available will report 0 bytes available on the socket but with sockets that doesn't mean there isn't more data on the way.

Correct. Available is simply reporting how many bytes are waiting to be read at that moment. Just keep reading until Receive() tells you that a disconnect has actually occurred. If you want to poll the socket status before calling Receive(), use Poll(), a disconnect will be reported as a readable state. Either way, Receive() will return 0 bytes on a graceful disconnect, and throw an exception on an abnormal disconnect.

Reading the FTP spec it seems as though the FTP server is supposed to shut down the data connection after it finishes sending all the data

In STREAM mode, yes.

but I don't think that is happening.

Yes, it is.

I can keep looping reading 0 bytes of data all day long. Am I looking at a broken FTP server or am I doing something wrong in my methodology?

You are doing something wrong in your methodology.

Using Filezilla, connected to the same FTP server, I see that when requesting a file the FTP server appends the file length to the 150 command in parenthesis. For example, "150 Opening BINARY mode data connection for test.zip (123421 bytes)." Is this text part of the FTP standard?

No, it is not. The text is arbitrary, it can be anything the server wants. What is important is the response code (150) to indicate the transfer is proceeding.

If all FTP servers use this same behavior and formatting

They don't use the same text formatting.

then I can just keep reading until I hit 123,421 bytes.

You are supposed to keep reading until the server closes the data connection. And then read the final response on the communication port to make sure the server considers the transfer to be successful, and not aborted prematurely.

Answer 2

It seems I did not read the spec well enough as there is a SIZE command in RFC 3659 that will return the size of the file. I tested it and it will do exactly what I need.