Persisting Data in a Twisted App

Question

I'm trying to understand how to persist data in a Twisted application. Let's say I've decided to write a Twisted server that:

1. Accepts inbound SMTP requests
2. Sends the message to a 3rd party system for modification
3. Relays the modified message to its destination

A typical Twisted tutorial would have you build this app using Deferreds and callbacks, roughly:

• A Factory handles inbound requests
• Each time a full email is received a call is sent to the remote message processor, returning a deferred
• Add an errback that substitutes the original message if anything goes wrong in the modify call.
• Add a callback to send the message on to the recipient, which again returns a deferred.
• A real server would add/include additional call/errbacks to retry or notify the sender or whatnot. Again for simplicity, assume we consider this an acceptable amount of effort and just log errors.

Of course, this persists NO data in the event of a crash/restart/something else. I get that a solution involves a 3rd party persistent datastore (RabbitMQ is often mentioned) and could probably come up with a dozen random ways to achieve the outcome.

However, I imagine there are a few approaches that work best in a Twisted app. What do they look like? How do they store (and restore in the event of a crash) the in-process messages?

Answer 1

If you found this question, you probably already know that Twisted is event-based. It sounds simple, but the "hardest" part of the answer is to get the persistence platform generating the events we need when we need them. Naturally, you can persist the data in a DB or a message queue, but some platforms don't naturally generate events. For example:

• ZeroMQ has (or at least had) no callback for new data. It's also relatively poor at persistence.

In other cases, events are easy but reliability is a problem:

• pgSQL can be configured to generate events using triggers, but they're one-time things so you can't resume incomplete events

The light at the end of the tunnel seems to be something like RabbitMQ.

• RabbitMQ can persist the message in a database to survive a crash
• We can use acknowledgements on both legs (incoming SMTP to RabbitMQ and RabbitMQ to outgoing SMTP) to ensure the application is reliable. Importantly, RabbitMQ supports acknowledgements.
• Finally, several of the RabbitMQ clients provide full asynchronous support (see for example pika, txampq, and puka)

It's enough for our purposes that the RabbitMQ client provides us an event-based interface.

At a more theoretical level, however, this need not be the case. In fact, despite the "notification" issue above, ZeroMQ has an event-based client. Even if our software is elegantly event-based, we will run into systems that aren't. In these cases, we have no choice but to fall back on polling. In principle, if not in practice, we just query the message provider for messages. When we exhaust the current queue (and immediately if there are no messages), we use a callLater command to check again in the future. It may feel anti-pattern, but it's (to the best of my knowledge anyway) the right way to handle this particular case.