ant ftp doesn't download files in subdirectories

Question

I'm trying to download files in subdirectories from an ftp server with ant. The exact set of files is known. Some of them are in subdirectories. Ant only seems to download the ones in the root directory. It does work if I download all files without listing them.

The first ftp action should do the exact same thing as the second. Instead I get "Hidden file \\a\a.txt assumed to not be a symlink."

Does anyone know what's wrong here? Is this a bug in the ant FTP task?

<?xml version="1.0" encoding="utf-8"?>
<project name="example" default="example" basedir=".">
    <taskdef name="ftp" 
    classname="org.apache.tools.ant.taskdefs.optional.net.FTP" />

    <target name="example">

        <!-- doesn't work -->
        <ftp action="get" verbose="true"
        server="localhost" userid="example" password="example" 
        remotedir="">
            <fileset dir="downloads" casesensitive="false" 
            includes="a/a.txt,a/b/ab.txt,c/c.txt" />
        </ftp>

        <!-- works (but requires multiple ftp tasks) -->
        <ftp action="get" verbose="true"
        server="localhost" userid="example" password="example"
        remotedir="a">
            <fileset dir="downloads" casesensitive="false" 
            includes="a.txt,b/ab.txt" />
        </ftp>
        <ftp action="get" verbose="true"
        server="localhost" userid="example" password="example"
        remotedir="c">
            <fileset dir="downloads" casesensitive="false" 
            includes="c.txt" />
        </ftp>

    </target>

</project>

Update: I posted a bug about this to Commons Net jira https://issues.apache.org/jira/browse/NET-324

Update: I added a bugreport to the ant bugreport system https://issues.apache.org/bugzilla/show_bug.cgi?id=49296

Answer 1

I believe to have found and fixed the problem. Also see my the bugreport in jira: https://issues.apache.org/jira/browse/NET-324

and I have added a bugreport to the ant bugreport system https://issues.apache.org/bugzilla/show_bug.cgi?id=49296

diff --git a/libs/apache-ant-1.8.1/src/main/org/apache/tools/ant/taskdefs/optional/net/FTP.java b/libs/apache-ant-1.8.1/src/main/org/apache/tools/ant/taskdefs/optional/net/FTP.java
--- a/libs/apache-ant-1.8.1/src/main/org/apache/tools/ant/taskdefs/optional/net/FTP.java
+++ b/libs/apache-ant-1.8.1/src/main/org/apache/tools/ant/taskdefs/optional/net/FTP.java
@@ -834,8 +834,7 @@
                         } else if (!result) {
                             return;
                         }
-                        this.curpwd = this.curpwd + remoteFileSep
-                            + currentPathElement;
+                        this.curpwd = getCurpwdPlusFileSep() + currentPathElement;
                     } catch (IOException ioe) {
                         throw new BuildException("could not change working dir to "
                                                  + (String) pathElements.elementAt(fcount)
@@ -895,7 +894,7 @@
              * @return absolute path as string
              */
             public String getAbsolutePath() {
-                return curpwd + remoteFileSep + ftpFile.getName();
+                return getCurpwdPlusFileSep() + ftpFile.getName();
             }
             /**
              * find out the relative path assuming that the path used to construct
@@ -1036,6 +1035,17 @@
             public String getCurpwd() {
                 return curpwd;
             }
+
+            /**
+             * @return parent directory of the AntFTPFile with a remoteFileSep appended
+             */
+            private String getCurpwdPlusFileSep() {
+                if (this.curpwd.endsWith(remoteFileSep)) {
+                    return this.curpwd;
+                }
+                return this.curpwd + remoteFileSep;
+            }
+            
             /**
              * find out if a symbolic link is encountered in the relative path of this file
              * from rootPath.

Answer 2

As I need this to work, I've made this workaround. It seems to work, but I'm not entirely pleased with it. It doesn't feel clean.

<scriptdef name="my-ftp-get" language="javascript">
    <attribute name="server"/>
    <attribute name="userid"/>
    <attribute name="password"/>
    <attribute name="remotedir"/>
    <attribute name="fileset_dir"/>
    <attribute name="fileset_includes"/>
    <![CDATA[
    importClass(java.io.File);
    importClass(org.apache.tools.ant.taskdefs.optional.net.FTP);
    var local_basedir = "" + attributes.get("fileset_dir") + "/";
    var original_includes = "" + attributes.get("fileset_includes");
    var remotedir = "" + attributes.get("remotedir");
    local_basedir = local_basedir.replace(/\\/g, "/");
    original_includes = original_includes.replace(/\\/g, "/");
    remotedir = remotedir.replace(/\\/g, "/");
    var includes_arr = original_includes.split(",");
    var clean_includes = {};
    for (var i = 0; i < includes_arr.length; i++) {
        var directory = "/";
        var filename = includes_arr[i];
        var split_include = includes_arr[i].split("/");
        if (split_include.length > 1) {
            directory = split_include[0] + "/";
            filename = includes_arr[i].substring(directory.length);
        }
        if (!clean_includes.hasOwnProperty(directory)) {
            clean_includes[directory] = [];
        }
        clean_includes[directory].push(filename);
    }
    var get_files = new FTP.Action();
    get_files.setValue("get");
    for (var path in clean_includes) {
        var current_clean_includes = clean_includes[path].join(",");
        var fileset = project.createDataType("fileset");
        var ftp = self.project.createTask("ftp");
        ftp.setAction(get_files);
        ftp.setServer(attributes.get("server"));
        ftp.setUserid(attributes.get("userid"));
        ftp.setPassword(attributes.get("password"));
        ftp.setRemotedir(remotedir + path);
        fileset.setDir(new File(local_basedir + path));
        fileset.setIncludes(current_clean_includes);
        ftp.addFileset(fileset);
        ftp.perform();
    }
    ]]>
</scriptdef>

<my-ftp-get
server="localhost" userid="example" password="example"
remotedir=""
    fileset_dir="downloads" casesensitive="false" 
    fileset_includes="a/a.txt,a/b/ab.txt">
</my-ftp-get>

Answer 3

Did you already try to set '/' as value of remotedir?

<ftp action="get" verbose="true"
   server="localhost" userid="example" password="example" 
   remotedir="/">
  <fileset dir="downloads" casesensitive="false" 
    includes="a/a.txt" />
</ftp>

Answer 4

I don't know that setting remotedir="" is legitimate. If you want to use the default root directory then you should leave it out entirely.

Answer 5

The first case will try to get the file //downloads/a/a.txt The second case will try to get the file //a/downloads/a.txt

If you replace the one that doesn't work with

<ftp action="get" verbose="true"
    server="localhost" userid="example" password="example" 
    remotedir="">
        <fileset dir="a/downloads" casesensitive="false" 
        includes="a.txt" />
</ftp>

or

<ftp action="get" verbose="true"
    server="localhost" userid="example" password="example" 
    remotedir="">
        <fileset dir="a" casesensitive="false" 
        includes="downloads/a.txt" />
</ftp>

it should work better. If these do not suit your needs could you provide the full path of the file(s) you want to match and maybe we can create some alternatives.

Answer 6

You may have more success with the Apache Commons VFS Ant Tasks. These provide reading/writing to various file systems, with copying. synchronization and other operations. This will also be a flexible solution should you need to change to use sftp, ssh, http or some other system later; unlike the native ant tasks for ftp, ssh etc, the majority of the VFS configuration is independent of filesystem, with filesystem specifics mostly requiring just a change of URL.

Your example would be coded along these lines:

<taskdef resource="org/apache/commons/vfs/tasks/tasks.properties"/>

<target name="example">
     <v-copy destdir="${basedir}" srcdir="ftp://example:example@localhost/downloads" includes="a/a.txt,a/b/ab.txt,c/c.txt"/>
</target>