Handling parallel ftp download errors

Question

I am downloading multiple files from a ftp server simultaneously using multiple sessions of Python. At some point one session (I suspect) reads a file that is being accessed by another process and throws the following error:

Traceback (most recent call last):   File
"F:\utilities\python\downloadFTP_NV.py", line 66, in <module>
     os.unlink(FILE) WindowsError: [Error 32] The process cannot access the file because it is being used by another process:
u'm_4011851_ne_11_1_20100620.tif'

This is the block of code where I cannot seem to find the best way to handle the errors and move to the next file to download:

        # Set logic so that already downloaded or partially
        # downloaded files will not be downloaded again
        if os.path.exists(fileCheck): # "fileCheck" is a file prior to renaming
            print 'File "%s" exists already' % name
            pass

        elif os.path.exists(fileCheck2): # "fileCheck2 is a file after renaming
            print 'File "%s" exists already' % FILE
            pass

        else:
            try:
                f.cwd(DIRN + folder)
                start = time.clock()
                f.retrbinary('RETR %s' % FILE, open(FILE, 'wb').write)
                end = time.clock()
                arcpy.Rename_management(os.path.join(workspace, FILE), os.path.join(workspace, name))
            except ftplib.error_perm:
                print 'ERROR: cannot read file "%s"' % FILE
                os.unlink(FILE)

I have thought about adding another except statement with a time.sleep(5) to help reduce overlapping processes. Or perhaps simply removing the os.unlink(FILE) line. What is the best method to handle errors such as this one?

Answer 1

I know that this post is 3 years old, but for other users sake that could maybe have the same issue, i will give a possible solution In your code i don't see that you close your file after the operation, so i think that is where the issue is. As a best practice it is always good to use the with statement as it gives a context manager and a better error handling.

so your part of code where you retrieve a file should be like this

with open(FILE, 'wb') as fhandle:
      ftp.retrbinary('RETR ' + FILE, fhandle.write)

That should fix your problem. And you don't need the os.unlink(FILE)

If you need some background infos about the with statement, this is the place to visit PEP 343 -- The "with" Statement

Answer 2

I would say that problem is in the way how you launch the sessions and allow them to overlap the files. Try to run them within one master session and and pass files to download as parameters.

Or you can just use multiprocessing module which may be even easier.

I guess during paraller download multiple files you should not have case that two processes are tring to operate on same file (as long as you don't download in paraller way one file).