high event processing in twisted

Question

I need to write an app that can receive, process, and then send events at ~15k Events Per Second (EPS). I've been learning twisted and have been using it to benchmark some tests:

Twisted RX only = ~90K EPS
Twisted RX and TX = ~45K EPS (basically half of RX only)
Twisted RX, processing, and TX = ~6K EPS (close, but not 15K EPS)

The processing portion is mostly a single regex matches condition - a task that is CPU-bound. I tried using threads.deferToThread and callbacks but, as expected, didn't improve the CPU-bound processing.

My server has 256 cores and I'd love to be able to put them to use while using twisted. Can I wrap multiprocessing in with twisted? Each process needs to share a dict, so I'd have to use multiprocessing.Manager().

Can multiprocessing be done within twisted? Is there a faster way run CPU-heavy tasks (regex expressions) in parallel within twisted?

Answer 1

Like the commenter, I would point at Glyph's answer for the multiprocessing question.

With that you could spawn off a fleet of blocking regex matching processes and communicate with them via the childFDs IProcessProtocol.childDataReceived and IProcessTransport.writeToChild methods.

This would let your twisted reactor continue to run at full speed and should get you a lot closer to your non-processing numbers (minus the cpu time for managing the extra file descriptors (though that should be tiny as compared letting the regex block the reactor))