Python Twisted: separate protocol from processing

Question

I would like to implement an own TCP-based protocol to be used in a server on top of Twisted. The mechanics of such an protocol implementation are clear (inherit from Protocol and overwrite four inherited methods, build Factory).

However, I want my protocol to be separate from the application logic, in a way that:

• protocol: receives data from clients, decodes the bytestream and populates Python data structures, takes data structures, encodes to bytestream (text-based) and responds to clients

• application logic: receives said data structures, evaluates and return response data structures

How would I build a Twisted application in a way that neither depends on the other (i.e. loose coupling)? I would imagine that the protocol class would be instantiated with an application logic callback as parameter?

Edit:

In the meantime, I have this:

from twisted.internet.protocol import Protocol
from twisted.internet.protocol import Factory
from twisted.internet.endpoints import TCP4ServerEndpoint
from twisted.internet import reactor

class Logic:
    def process(self, data):
        return "Processed: %s" % data

class LineProtocol(Protocol):
    def dataReceived(self, data):
        print "Received: %s" % data
        if self.factory._logic_callback:
            ret = self.factory._logic_callback.process(data)
            print "Sent: %s" % ret
            self.transport.write(ret)

class LineFactory(Factory):
    protocol = LineProtocol

    def __init__(self, logic_callback = None):
        if logic_callback:
            self._logic_callback = logic_callback()
        else:
            self._logic_callback = None

endpoint = TCP4ServerEndpoint(reactor, 1234)
endpoint.listen(LineFactory(Logic))
reactor.run()

Would you consider this the "twisted" way? Anything to improve?

The code creates a Logic instance inside the LineFactory. Good to put it there?

Answer 1

Notice that you already have one example of loose coupling in your plan. The transport, which handles the implementation details of TCP, is separate from your protocol and the two objects interact through a well-defined interface: dataReceived et al on the protocol, write et al on the transport.

Just extend this idea another step. Have your application object and the protocol interact through a well-defined interface. Whether the application object is given to the protocol as an __init__ argument is somewhat up to what the expected interactions will be. If the application is purely reactive (ie, it just reacts to things that happen on the protocol, it doesn't initiate things itself) then this is a good approach. If the application is more interested in driving the action (for example, consider an application driving an HTTP client protocol: nothing at all happens on the network until the application decides it wants to send a request) then you may want to clearly define an interface for your protocol to have that the application is expected to use (for example, an HTTP client protocol might have a request method that takes a URL, an HTTP method, etc).

Keep in mind that there are some good APIs that give you back a newly protocol instance. For example:

from sys import argv

from twisted.internet.protocol import Protocol
from twisted.internet.endpoints import connectProtocol, clientFromString
from twisted.internet.task import react

def main(reactor, description):
    endpoint = clientFromString(reactor, description)
    connecting = connectProtocol(endpoint, Protocol())
    def connected(protocol):
        # Now `protocol` is connected somewhere and you can start
        # calling useful methods on it, if it has any.
        ...
    connecting.addCallback(connected)
    return connecting

react(main, argv[1:])

Put some useful code into connected, replace Protocol() with an instance of your more interesting protocol class, and run with an argument like "tcp:localhost:25".

From your example:

def dataReceived(self, data):
    print "Received: %s" % data
    if self.factory._logic_callback:
        ret = self.factory._logic_callback.process(data)
        print "Sent: %s" % ret
        self.transport.write(ret)

This isn't really beneficial. You haven't actually implemented any protocol logic in your protocol. All you've done is make self.factory._logic_callback responsible for implementing the protocol. It's a couple extra objects for no real benefit. I would not suggest doing things this way.

You want to do all parsing and serialization in your protocol class. Only delegate higher-level logic, implemented in terms of structured objects (the output of whatever parsing you do), to things other than the protocol.