Apache FTPClient - incomplete file retrieval on Linux, works on Windows

Question

I have a java application on Websphere that is using Apache Commons FTPClient to retrieve files from a Windows server via FTP. When I deploy the application to Websphere running in a Windows environment, I am able to retrieve all of the files cleanly. However, when I deploy the same application to Webpshere on Linux, there are cases where I am getting an incomplete or corrupt files. These cases are consistent though, such that the same files will fail every time and give back the same number of bytes (usually just a few bytes less than what I should be getting). I would say that I can read approximately 95% of the files successfully on Linux.

Here's the relevant code...

ftpc = new FTPClient();
// set the timeout to 30 seconds
    ftpc.enterLocalPassiveMode();
    ftpc.setDefaultTimeout(30000);
    ftpc.setDataTimeout(30000); 
    try
    {
              String ftpServer = CoreApplication.getProperty("ftp.server");
        String ftpUserID = CoreApplication.getProperty("ftp.userid");
        String ftpPassword = CoreApplication.getProperty("ftp.password");

        log.debug("attempting to connect to ftp server = "+ftpServer);
        log.debug("credentials = "+ftpUserID+"/"+ftpPassword);

        ftpc.connect(ftpServer);

        boolean login = ftpc.login(ftpUserID, ftpPassword);

        if (login) 
        {
            log.debug("Login success...");          } 
        else 
        {
            log.error("Login failed - connecting to FTP server = "+ftpServer+", with credentials "+ftpUserID+"/"+ftpPassword);
            throw new Exception("Login failed - connecting to FTP server = "+ftpServer+", with credentials "+ftpUserID+"/"+ftpPassword);
        }

        is = ftpc.retrieveFileStream(fileName);
          ByteArrayOutputStream out = null; 
          try { 

              out = new ByteArrayOutputStream(); 
              IOUtils.copy(is, out); 
          } finally { 
              IOUtils.closeQuietly(is); 
              IOUtils.closeQuietly(out); 
          }

          byte[] bytes = out.toByteArray();
          log.info("got bytes from input stream - byte[] size is "+ bytes.length);

Any assistance with this would be greatly appreciated.

Thanks.

Answer 1

I have a suspicion that the FTP might be using ASCII rather than binary transfer mode, and mapping what it thinks are Window end-of-line sequences in the files to Unix end-of-lines. For files that are really text, this will work. For files that are really binary, the result will be corruption and a slightly shorter file if the file contains certain sequences of bytes.

See FTPClient.setFileType(...).

FOLLOWUP

... so why this would work on Windows and not Linux remains a mystery for another day.

The mystery is easy to explain. You were FTP'ing files from a Windows machine to a Windows machine, so there was no need to change the end-of-line markers.