Question

I'm trying to upload 30+ files via ftplib to a webserver but I'm having issues with uploading taking too long.

If I use programs like CuteFTP, FTPPro or upload directly via the cpanel on my website it takes a few seconds to upload all 30+ files, however with this code regardless of my internet upload speed it takes 100~ seconds every time. I've tried it on three different connections in different countries.

8meg down, 8meg up connection took 102 seconds 8meg down, 0.5meg up connection took 106 seconds 8meg down, 1meg up connection took 103 seconds 8meg down, 0.8meg up connection took 101 seconds

Ideally I'd like to be able to zip all the files, upload them and extract them on the webserver.

The following code is some test code which uploads 30+ files all out of the same folder and 1 single file out of a different folder.

import glob
import os
import ftplib
import re
from time import time

starttime = time()
print "Running..."

## Create session
ftp_session = ftplib.FTP('ftp.website.com','admin@website.com','password123','',60)

## ----------------------------------------------------------------------------------------------------

## Output location for single file
destdir = '/MAIN/'
ftp_session.cwd(destdir)

## Stats & Uploading for single file
ftp_file = open("output.html",'rb')
size = os.stat("output.html").st_size / 1024
result = ftp_session.storbinary('STOR output.html.tmp', ftp_file)
ftp_file.close()
data1 = re.findall('(\d*\.\d*)\sseconds', result)
data2 = re.findall('(\d*\.\d*)\sKbytes\sper\ssecond', result)

## Prints and renaming for single file
print "Uploaded output.html (" + str(size) + "KB) in " + str(data1).strip("['").strip("']") + " seconds at " + str(data2).strip("['").strip("']") + " kbps"
if 199 < int(result[0:3]) < 300: # Positive Completion
    ftp_session.rename("output.html.tmp", "index.php")
else:
    print "Upload of output.html resulted in code %s" % (result)

## ----------------------------------------------------------------------------------------------------

## Have to use the full directory for some reason when really they're in \CLASSES\
sourcedir = 'C:\POL\OUTPUT\CLASSES'
filelist = glob.glob(os.path.join(sourcedir, '*.html'))

## Output location for multiple files
destdir = '/MAIN/CLASSES/'
ftp_session.cwd(destdir)

## Cycle through each file
for fname in filelist:
    filename = fname.replace(sourcedir+'\\', "")
    ## Ignore Classes0.html as we dont want to upload this file
    if filename != "CLASSES0.html":
        ## Stats & Uploading for multiple files
        ftp_file = open(fname,'rb')
        size = os.stat(fname).st_size / 1024
        result = ftp_session.storbinary('STOR %s.tmp' % filename, ftp_file)
        ftp_file.close()
        data1 = re.findall('(\d*\.\d*)\sseconds', result)
        data2 = re.findall('(\d*\.\d*)\sKbytes\sper\ssecond', result)

        ## Prints and renaming for multiple files
        print "Uploaded " + filename + " (" + str(size) + "KB) in " + str(data1).strip("['").strip("']") + " seconds at " + str(data2).strip("['").strip("']") + " kbps"
        if 199 < int(result[0:3]) < 300: # Positive Completion
            ftp_session.rename("%s.tmp" % filename, filename)
        else:
            print "Upload of %s resulted in code %s" % (filename, result)
            break

## ----------------------------------------------------------------------------------------------------

## Print end timer
print """The whole routine took %.3f seconds""" % (time() - starttime)

The output of the above code is:

Running...
Uploaded output.html (505KB) in 7.528 seconds at 67.13 kbps
Uploaded FULL.html (498KB) in 7.265 seconds at 68.55 kbps
Uploaded CLASSES1.html (14KB) in 0.697 seconds at 21.09 kbps
Uploaded CLASSES10.html (14KB) in 0.625 seconds at 23.93 kbps
Uploaded CLASSES11.html (14KB) in 0.788 seconds at 18.78 kbps
Uploaded CLASSES12.html (12KB) in 0.668 seconds at 18.87 kbps
Uploaded CLASSES13.html (12KB) in 0.631 seconds at 20.02 kbps
Uploaded CLASSES14.html (20KB) in 0.815 seconds at 25.21 kbps
Uploaded CLASSES15.html (13KB) in 0.633 seconds at 21.73 kbps
Uploaded CLASSES16.html (20KB) in 0.876 seconds at 23.25 kbps
Uploaded CLASSES17.html (11KB) in 0.675 seconds at 16.97 kbps
Uploaded CLASSES18.html (13KB) in 0.618 seconds at 21.97 kbps
Uploaded CLASSES19.html (20KB) in 0.830 seconds at 24.90 kbps
Uploaded CLASSES2.html (12KB) in 0.610 seconds at 20.62 kbps
Uploaded CLASSES20.html (11KB) in 0.611 seconds at 18.77 kbps
Uploaded CLASSES21.html (26KB) in 0.818 seconds at 32.05 kbps
Uploaded CLASSES22.html (10KB) in 0.646 seconds at 16.00 kbps
Uploaded CLASSES23.html (13KB) in 0.652 seconds at 21.04 kbps
Uploaded CLASSES24.html (22KB) in 0.839 seconds at 27.11 kbps
Uploaded CLASSES25.html (13KB) in 0.627 seconds at 21.85 kbps
Uploaded CLASSES26.html (20KB) in 0.830 seconds at 24.71 kbps
Uploaded CLASSES27.html (11KB) in 0.627 seconds at 18.96 kbps
Uploaded CLASSES28.html (12KB) in 0.612 seconds at 20.43 kbps
Uploaded CLASSES29.html (16KB) in 0.625 seconds at 25.76 kbps
Uploaded CLASSES3.html (11KB) in 0.632 seconds at 17.93 kbps
Uploaded CLASSES30.html (14KB) in 0.636 seconds at 22.93 kbps
Uploaded CLASSES31.html (16KB) in 0.629 seconds at 26.03 kbps
Uploaded CLASSES32.html (19KB) in 0.636 seconds at 30.93 kbps
Uploaded CLASSES33.html (20KB) in 0.869 seconds at 23.69 kbps
Uploaded CLASSES34.html (20KB) in 0.830 seconds at 24.63 kbps
Uploaded CLASSES35.html (20KB) in 0.822 seconds at 25.06 kbps
Uploaded CLASSES36.html (18KB) in 0.647 seconds at 28.32 kbps
Uploaded CLASSES37.html (15KB) in 0.639 seconds at 24.63 kbps
Uploaded CLASSES38.html (16KB) in 0.630 seconds at 25.50 kbps
Uploaded CLASSES39.html (18KB) in 0.614 seconds at 29.76 kbps
Uploaded CLASSES4.html (14KB) in 0.682 seconds at 21.65 kbps
Uploaded CLASSES40.html (17KB) in 0.715 seconds at 24.08 kbps
Uploaded CLASSES41.html (19KB) in 0.690 seconds at 28.17 kbps
Uploaded CLASSES42.html (17KB) in 0.686 seconds at 25.11 kbps
Uploaded CLASSES5.html (11KB) in 0.625 seconds at 18.22 kbps
Uploaded CLASSES6.html (14KB) in 0.649 seconds at 22.91 kbps
Uploaded CLASSES7.html (25KB) in 0.922 seconds at 28.14 kbps
Uploaded CLASSES8.html (16KB) in 0.676 seconds at 23.70 kbps
Uploaded CLASSES9.html (13KB) in 0.697 seconds at 19.46 kbps
The whole routine took 102.950 seconds

Any ideas on how to make this significantly faster?

No correct solution

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top