pyftpdlib slow .read on file blocks entire mainloop

Question

Helllo,

I am using a custom AbstractFS on pyftpdlib that maps files on a HTTP server to FTP. This files are returned by my implementation of open (of AbstractFS) which returns a httplib.HTTPResponse wrapped by the following class:

class HTTPConnWrapper:
    def __init__(self, obj, filename):
        # make it more file obj like
        self.obj = obj
        self.closed = True
        self.name = filename.split(os.sep)[-1]

    def seek(self, arg):
        pass

    def read(self, bytes):
        #print 'read', bytes
        read = self.obj.read(100) #we DONT read var byes, but 100 bytes
        #print 'ok'
        return read

The problem is that if a client is downloading files the entire server becommes sluggish. What can I do? Any ideas?

PS: And why just monkey patching everything with evenetlet does'nt magically makes everything work?

Answer 1

Ok, I posted a bug report on pyftpdlib:

I wouldn't even know what to recommend exactly as it's a problem which is hard to resolve and there's no easy or standard way to deal with it.

But I got a crazy solution to solve this problem without using pyftpdlib.

1. rewrite everything using wsgidav (which uses the cherrypy wsgiserver, so its threaded)
2. mount that WebDAV filesystem as native filesystem (net use on windows, mount.davfs on linux)
3. serve this mounted filesystem with any ftp server that can handle blocking file systems

Answer 2

pyftpdlib uses Python's asyncore module which polls and interacts with dispatchers. Each time you map an FTP request to a request to the HTTP server you're blocking the asyncore loop that pydftpdlib is using. You should implement your HTTP requests as dispatchers that fit the asyncore model, or generate threads to handle the request asynchronously and post the result back to the FTP request handler when the data has arrived. This is somewhat difficult as there's no provided mechanism to interrupt asyncore's polling loop from external threads.

As for eventlet, I don't know that it would play nicely with asyncore, which is already utilizing a nonblocking IO mechanism.