Traversing FTP listing

Question

I am trying to to get all directories' name from an FTP server and store them in hierarchical order in a multidimensional list or dict

So for example, a server that contains the following structure:

/www/
    mysite.com
        images
            png
            jpg

at the end of the script, would give me a list such as

['/www/'
  ['mysite.com'
    ['images'
      ['png'],
      ['jpg']
    ]
  ]
]

I have tried using a recursive function like so: def traverse(dir): FTP.dir(dir, traverse)

FTP.dir returns lines in this format:

drwxr-xr-x    5 leavesc1 leavesc1     4096 Nov 29 20:52 mysite.com

so doing line[56:] will give me just the directory name(mysite.com). I use this in the recursive function.

But i cannot get it to work. I've tried many different approaches and can't get it to work. Lots of FTP errors as well (either can't find the directory - which is a logical issue, and sometimes unexpected errors returned by the server, which leaves no log and i can't debug)

bottom line question: How to get a hierarchical directory listing from an FTP server?

Answer 1

Here is a naive and slow implementation. It is slow because it tries to CWD to each directory entry to determine if it is a directory or a file, but this works. One could optimize it by parsing LIST command output, but this is strongly server-implementation dependent.

import ftplib

def traverse(ftp, depth=0):
    """
    return a recursive listing of an ftp server contents (starting
    from the current directory)

    listing is returned as a recursive dictionary, where each key
    contains a contents of the subdirectory or None if it corresponds
    to a file.

    @param ftp: ftplib.FTP object
    """
    if depth > 10:
        return ['depth > 10']
    level = {}
    for entry in (path for path in ftp.nlst() if path not in ('.', '..')):
        try:
            ftp.cwd(entry)
            level[entry] = traverse(ftp, depth+1)
            ftp.cwd('..')
        except ftplib.error_perm:
            level[entry] = None
    return level

def main():
    ftp = ftplib.FTP("localhost")
    ftp.connect()
    ftp.login()
    ftp.set_pasv(True)

    print traverse(ftp)

if __name__ == '__main__':
    main()

Answer 2

Here's a first draft of a Python 3 script that worked for me. It's much faster than calling cwd(). Pass in server, port, directory, username, and password as arguments. I left output as a list as an exercise for the reader.

import ftplib
import sys

def ftp_walk(ftp, dir):
    dirs = []
    nondirs = []
    for item in ftp.mlsd(dir):
        if item[1]['type'] == 'dir':
            dirs.append(item[0])
        else:
            nondirs.append(item[0])
    if nondirs:
        print()
        print('{}:'.format(dir))
        print('\n'.join(sorted(nondirs)))
    else:
        # print(dir, 'is empty')
        pass
    for subdir in sorted(dirs):
        ftp_walk(ftp, '{}/{}'.format(dir, subdir))

ftp = ftplib.FTP()
ftp.connect(sys.argv[1], int(sys.argv[2]))
ftp.login(sys.argv[4], sys.argv[5])
ftp_walk(ftp, sys.argv[3])

Answer 3

You're not going to like this, but "it depends on the server" or, more accurately, "it depends on the output format of the server".

Different servers can be set to display different output, so your initial proposal is bound to failure in the general case.

The "naive and slow implementation" above will cause enough errors that some FTP servers will cut you off (which is probably what happened after about 7 of them...).

Answer 4

If the server supports the MLSD command, then use the “a directory and its descendants” code from that answer.

Answer 5

If we are using Python look at:

http://docs.python.org/library/os.path.html (os.path.walk)

If there already is a good module for this, don't reinvent the wheel. Can't believe the post two spots above got two ups, anyway, enjoy.