Python tornado can only open one page when processing

Question

I have a python code that upload files to another server when a button is click. After clicking the button, web server will start uploading file to another server that took a long process, during that time the the browser is loading, and I can't open another page of my website.

I try to use async, but async example all uses httpclient which I am not using, then I modify the code like below

The upload itself is a synchronous code, and I don't want to modify it, what I want is a wrapper or something that can allow the client to open another page and does not lock the session while processing. Is there such a way?

def Upload(self, filehost, uniqueno, username, password):
    Filename = self.GetFile(uniqueno)

    uploadfunc = None
    if (upload == "mega"):
        uploadfunc = MegaWrapper.Upload

    result = Thread(target=self.MyUpload, args=(uploadfunc, MyConfig.Folder + Filename, username, password))
    result.start()
    result.join(5 * 60)

    return self.UploadResult

def MyUpload(self, uploadfunc, filename, username, password):
    str = uploadfunc(filename, username, password)
    self.UploadResult = str
    return str

def post(self):
    filehost = self.get_argument("filehost")
    crc = self.get_argument("crc")
    username = self.get_argument("username")
    password = self.get_argument("password")

    fileList = dal.FileListByCRC(crc)
    messagetype = "error"
    mesasge = ""
    if (len(fileList) > 0):
        try:
            url = self.Upload(filehost, fileList[0].UniqueNo , username, password)
            message = url
            if (StringOps.StartWith(url, "http")):
                messagetype = "info"
                message = "Success Uploading : " + url
        except Exception, ex:
            messagetype = "error"
            message = "Fail Uploading : " + str(ex)    

    self.render("upload.html", title=MyConfig.DefaultSiteTitle + " - Administrative Page - Upload/Delete", messagetype=messagetype, message=message, crc=self.get_argument("crc"), filehost=self.get_argument("filehost"), username=self.get_argument("username"), password=self.get_argument("password"))

Answer 1

You basically have to write your own handler that also make use of the IOLoop correctly. Look into tornado.ioloop.IOLoop.add_callback and friends. That being said, this wasn't intuitive and took me some time to figure out how to do async downloading of large binary data, and at the end of it I concluded that this probably isn't the best tool for serving large binary data that exceed a typical transmission window.

Of course, you probably want to know how to start, here is something to help you get started:

import time
import logging
from datetime import timedelta
from threading import Thread
from tornado.httpserver import HTTPServer
import tornado.ioloop
import tornado.web

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)


def blocking_task(callback, seconds=10):
    time.sleep(seconds)
    callback('slept for %d seconds' % seconds)


class CallbackRequestHandler(tornado.web.RequestHandler):
    wait = timedelta(seconds=1)

    @tornado.web.asynchronous
    def get(self):
        tornado.ioloop.IOLoop.current().add_callback(self.start_loop)

    def start_loop(self):
        self.task = Thread(target=blocking_task,
            kwargs={'callback': self.write})
        self.task.start()
        logger.info('starting %s', self.task)
        self.loop()

    def loop(self):
        if self.task.is_alive():
            logger.info('waiting on %s', self.task)
            tornado.ioloop.IOLoop.current().add_timeout(self.wait, self.loop)           
        else:
            logger.info('finished on %s', self.task)
            self.finish()


tornado_app = tornado.web.Application([
    (r"/", CallbackRequestHandler),
])

if __name__ == "__main__":
    http_server = HTTPServer(tornado_app)
    http_server.listen(9090)
    tornado.ioloop.IOLoop.instance().start()

An example run with two clients poking at this instance at http://localhost:9090.

$ python unblock.py
INFO:__main__:starting <Thread(Thread-1, started 139876219111168)>
INFO:__main__:waiting on <Thread(Thread-1, started 139876219111168)>
INFO:__main__:waiting on <Thread(Thread-1, started 139876219111168)>
INFO:__main__:starting <Thread(Thread-2, started 139876210718464)>
INFO:__main__:waiting on <Thread(Thread-2, started 139876210718464)>
INFO:__main__:waiting on <Thread(Thread-1, started 139876219111168)>
INFO:__main__:waiting on <Thread(Thread-2, started 139876210718464)>
INFO:__main__:waiting on <Thread(Thread-1, started 139876219111168)>
INFO:__main__:waiting on <Thread(Thread-2, started 139876210718464)>
INFO:__main__:waiting on <Thread(Thread-1, started 139876219111168)>
INFO:__main__:waiting on <Thread(Thread-2, started 139876210718464)>
INFO:__main__:waiting on <Thread(Thread-1, started 139876219111168)>
INFO:__main__:waiting on <Thread(Thread-2, started 139876210718464)>
INFO:__main__:waiting on <Thread(Thread-1, started 139876219111168)>
INFO:__main__:waiting on <Thread(Thread-2, started 139876210718464)>
INFO:__main__:waiting on <Thread(Thread-1, started 139876219111168)>
INFO:__main__:waiting on <Thread(Thread-2, started 139876210718464)>
INFO:__main__:waiting on <Thread(Thread-1, started 139876219111168)>
INFO:__main__:waiting on <Thread(Thread-2, started 139876210718464)>
INFO:__main__:waiting on <Thread(Thread-1, started 139876219111168)>
INFO:__main__:waiting on <Thread(Thread-2, started 139876210718464)>
INFO:__main__:finished on <Thread(Thread-1, stopped 139876219111168)>
INFO:tornado.access:200 GET / (::1) 10010.21ms
INFO:__main__:waiting on <Thread(Thread-2, started 139876210718464)>
INFO:__main__:finished on <Thread(Thread-2, stopped 139876210718464)>
INFO:tornado.access:200 GET / (::1) 10004.87ms

This is probably NOT the optimum solution as I don't make use of the newer/better methods such as concurrent.futures and the related hooks that Tornado provides, but hopefully this example code will point you towards the right direction.