How can I use Jinja with Twisted?

Question

I'm planning up a discussion software using Python with Twisted, Storm, and Jinja. The problem is that Jinja was not made for Twisted or asynchronous socket libraries, and the performance provided by using Twisted is why I don't plan on using Flask.

So, how can I have Twisted render webpages using Jinja?

Answer 1

You can render web pages using Jinja the same way you would use any other Python library in Twisted. You just call into it. This will work fine with Twisted, although you may run into performance issues if Jinja does something blocking. Note that it is possible to use blocking libraries with Twisted just fine, either via deferToThread or just blocking the mainloop if it's not a performance problem. So, I am inferring that your question is really about how to use Jinja without blocking.

Jinja is a templating library, which means that it reads a template, invokes some view logic on the template, and writes some HTML output. So there are 3 things that can block:

1. reading the template,
2. writing the result.
3. running the view logic (your application code),

I don't know Jinja, so I don't know exactly how the APIs for each of these things is structured and I can't tell you what to do, but my guess is that that part's easy; so, I'm going to give you a general answer about 3rd-party templating libraries and Twisted.

So I'll address each of these concerns, although not quite in order:

1. Reading the Template

Really the most reasonable thing to do here is to not care about it. Reading the template is probably really fast. These are frequently-accessed, tiny files, which your operating system is almost certainly keeping in its filesystem cache. It's unlikely that you're ever going to block on reading them unless you're doing something nuts like putting them on NFS. If you profile your application and find that this is a problem – because, let's say, you have extremely slow disks or a remote filesystem – just read the template into a cStringIO or something similar at startup time and feed it to jinja after that.

3. Writing the Response

Web pages are not all that big, and Twisted doesn't provide a blocking API to write to sockets. Instead, it offers an API which just buffers the whole result in memory until it can be written out. My suggestion is to do basically the same thing here as with reading the template: unless you have seriously huge output, it's probably fine to burn up a little bit of RAM while the response is being fed to the client.

2. Running your View Logic

This is the area where you are most likely to run into problems. Jinja probably doesn't handle the results of Deferreds. But actually, it's not actually Jinja which is going to give you problems directly: it's Storm. Storm expects to be able to block, to make database queries, when you access certain attributes. Talking to the database blocks, and this is the most signifiant source of blocking I/O in most web applications. So you need to decide how you're going to deal with that. You have a few options:

1. Just do it in the main thread and don't worry about it. Maybe your application is for a workgroup of 10 people and your database is local. Sure, your I/O is going to block, but if it still meets its performance requirements, who cares? Not every application has to scale to the moon and back.
2. Pre-fetch everything from storm in a deferToThread call (or similar) and make sure Jinja is only accessing objects in memory. This way you can run your renderers in your main thread, in a callback on a Deferred that was doing database I/O. If you're only accessing objects in memory, your renderer still might take a little while, but that's fine. This question prompted me to post an article to my blog about the distinction between "blocking" and "running" which had been hanging out as a draft for quite a while; you may want to go read it.
3. Do your whole render in a thread or subprocess, and treat it as a blocking component of your program. This loses some of the benefits of using Twisted, but it's still a perfectly viable strategy to integrate a blocking Jinja/Storm component and a non-blocking pure-Twisted component, the actual chat-message relaying part.

If none of these options work for you, Twisted has included a templating library that does support Deferreds since version 11.0. You may consider using twisted.web.template as an alternative to Jinja.

Answer 2

Here is sample example on how to implement solution 3, with basic deferred returning function support:

from jinja2 import Template
from twisted.internet import threads, reactor, defer

def inThread(f):
    def new_f(*args, **kwargs):
        return threads.deferToThread(f, *args, **kwargs)
    return new_f


def fromThread(f):
    def new_f(*args, **kwargs):
        return threads.blockingCallFromThread(reactor, lambda: defer.maybeDeferred(f, *args, **kwargs))
    return new_f


class DeferredTemplate(Template):
    def render(self, **kw):
        hooked_kw = {}
        for k, v in kw.iteritems():
            # decorate the callable so that they are run in the main thread
            if callable(v):
                v = fromThread(v)
            hooked_kw[k] = v
        return inThread(Template.render)(self, **hooked_kw)

from twisted.trial import unittest
class TestJinjaDeferred(unittest.TestCase):
    @defer.inlineCallbacks
    def test_basic(self):
        def getHello():
            d = defer.Deferred()
            reactor.callLater(0.0, lambda: d.callback("Hello"))
            return d

        def getWorldSync():
            return "world"

        template = DeferredTemplate("{{ getHello() }} {{ getWorldSync() }}")
        res = yield template.render(getHello=getHello, getWorldSync=getWorldSync)
        self.assertEqual(u"Hello world", res)

Answer 3

i think Tornado template system (it's like the Jinja2 template since it's a Django-like...) can be used without the tornado itself:

We attempted to clean up the code base to reduce interdependencies between modules, so you should (theoretically) be able to use any of the modules independently in your project without using the whole package.

Tornado Templates