How do I approach polyglot plugins for asynchrous python applications?

Question

I've been looking at ways to let my app's users write plugins for it. However, to give them more options, I decided to implement a polyglot plugin system.

From the engineering perspective there are multiple ways to do this.

• Transpiling: Not many libraries that can reliably do this.
• Bindings: Types can be a pain.
• Using APIs: Seems like the easiest route, but plugins end up being complex.

One method I thought of was to use Jupyter kernels. Sending messages(via IPC) between event loops that already run in the respective interpreters. I only want to support interpreted/dynamic languages(at least 5) so it will be easier to load at run-time.

Are there are any common methods that I have missed out on and how do I go about building an application like this? What do I have to watch out to prevent the most common pitfalls?

All this has to work while still supporting some sort of event loop. However, I don't need threading since python doesn't have good support for it anyways.

Answer 1

Cross-language event loops aren't entirely impossible, but if you go that route you are in for a world of pain. This alone excludes some solutions such as autogenerated language bindings.

The only straightforward way to connect any two programming languages is through IPC. Let your application start plugins as subprocesses, and communicate via pipes. Alternatively, implement your application as a server that separate processes can connect to. Added benefit of IPC-based solutions: most event loops make it easy to trigger on readable I/O handles.

Over these pipes, you exchange messages of any format. This could be a simple “one JSON document per line” protocol, Protobuf messages, or full-on HTTP. This is the the part were you can choose a design that is easy to implement for all participants, both in regards to the message serialization technology and to the protocol implemented by these messages.

How the plugin protocol should be structured depends on what these applications are supposed to be doing. E.g. a plugin may be a filter that massages some data. Then, the main application has code like this:

sendMessageToPlugin(data)
result = await readMessageFromPlugin()

So this is effectively synchronous communication with the plugin, but doesn't block your application. The plugin must still respond to messages in order, but that may not be a problem for many workloads.

Another simple case is when the plugin is merely an event consumer. So you just send messages to the plugin without having to receive any responses. But this opens up some difficult questions: What if there are more events than the plugin can currently consume? How can they be buffered? Should they be buffered? What happens when the plugin crashes?

In these more complicated cases, it may be sensible to use an existing message queue that serves as a broker between your plugins and your application.

It is generally sensible to offer a small, simple, well-defined API to plugins. This makes it impossible for the plugin to request arbitrary data. Instead, you define messages that contain the necessary data. If you need a shared datamodel, using an external database may be sensible because it can do atomic updates, and ensures consistency of your datamodel. Your messages would then contain database keys, not the full data.

Note that the further your design evolves in some direction, the easier it gets to apply best practices from SOA or Microservice approaches. All of these address the same problem: language-agnostic cooperation between multiple processes.

I would also like to point out that these engineering challenges, while not unsolvable, do still require lots of effort. Unless you have a concrete need that justifies the cost of developing this plugin system, it may be better to stay within the host language. Integrating a library is so much easier than dealing with IPC.