Platform independant ways of passing messages between executibles

Question

I am working on an a data collector which could be called an API of some sorts, which is based on .Net Core 2.1 . It's job is to receive and pull (ask for) data from data collectors(executable, DLL,service), those can be in any language, but in this case let's stick with C++.

Collector (API) has to be platform independent, hence .NET Core (this is not going to change), while data collectors can be written in any language, run in any way they see fit, be platform dependant, but not limited to how many of them can send data to collector.

This is all local on same machine, possibly between different user accounts and different sessions due to security reasons.

My idea towards implementing this is to create ASP.NET Core web API which would be self hosted through kestrel locally. It's lightweight and does not require a proper server since it will never deal with outside traffic. Can run on Windows, Linux and Mac. Transport between collector and data collectors would be web/http based, which is available from pretty much any language and something that majority of developers known how to maintain.

But is it an overkill? Just to pass messages between processes on same machine, are there any alternatives? I come primary from web development background, so this is quite new to me.

In short, I need a Collector C, receive and ask for data from Data Collectors D. This can be through events, subscriptions, message passing, D calling C with data, C processes it. C calls D to ask if there is any data. D calls C with data (can be new call triggered by asking for data). C is platform independent, D can be platform dependant, but must be able to communicate with C. There can be many D, in C++, Java, C#, running in Windows, Linux, Mac. Collector C is alone, always same, can only receive and ask for data and is platform independent.

Are there any messaging frameworks apart from http that can be used to implement this?

Answer 1

Consider using a message queue (e.g. RabbitMQ). It will provide the following benefits:

1. A robust message transport. Order of delivery is guaranteed if all messanging is performed in the same queue. Messages are removed from the queue only after they have been fully processed and acknowledged by the consumer app.
2. Possibility to separate your modules over different machines. E.g. a docker container per module.
3. If one module goes down, the messages in it's input queue will be persisted for a long period of time (until the queue runs out of the limit of waiting messages). After coming back up again, the module will consume all the message history that it missed.
4. Backpressure out of the box: if messages come in bursts, sometimes faster then you process them, the message queue will provide a buffer which will not slow down the producer and will not cause consumer to overrun.
5. Potentially language- and platform-agnostic solution. Just implement the logic of consuming and producing messages of your module for the platform (and also serialization and deserialization, as Sean Burton has mentioned) - and you're good to go.

The downside of it is that you would have another technology in your stack.

Answer 2

But is it an overkill?

No, it's not necessarily overkill. If you want to transmit data between processes in a portable manner then you definitely should be using a well-defined API and serialisation method. A web interface that uses JSON serialisation is certainly one way of doing this. There are other options, such as using binary serialisation (Protocol Buffers, for example) and transmitting over TCP. Which of these is best for you depends on your requirements, for example a binary protocol is likely to be faster and more efficient (less bytes transferred), so you might want to consider that if you need very high performance.

Answer 3

You could consider using SignalR for asp.net core. It's designed for real time communication via RPC calls. It's not just for web apps.

Create a hub to act as the host and coordinator of calls and several clients that collect the data. It can do everything you mentioned in the last paragraph.