Communication between applications written in different languages

Question

I am looking at linking a few applications together (all written in different languages like C#, C++, Python) and I am not sure how to go about it.

What I mean by linking? The system I am working on consists of small programs each responsible for a particular processing task. I need to be able to transfer a data set from one application to another easily (the data set in question is not huge, probably a few megabytes) and I also need some form of way to control the current state of the operation (This is where a client-server model rings a bell)

It seems like sockets or maybe SOAP would be a universal solution but just wanted to get some opinions as to what people think about this subject.

Comments/suggestions will be appreciated, thanks!

Answer 1

I personally take a liking towards ØMQ. It's a library that has a familiar BSD-sockets-like interface for passing messages, but you'll find it implements interesting patterns for distributing tasks.

It sounds like you want to arrange several processes in a pipeline. ØMQ allows you to do that using push and poll sockets. (And afterwards, you'll find it's even possible to scale up across multiple processes and machines with little effort.) Take a look at the guide to get started, and the zmq_socket(3) manpage specifically for how push and pull works.

Bindings are available for all the languages you mention.

As for the contents of the message, ØMQ doesn't concern itself with that, they are just blocks of raw data. You can use any format that suits you, such as JSON, or perhaps Protocol Buffers.

What I'm not sure about is the ‘controlling state’ you mention. Are you interested in, for example, cancelling a job halfway through?

Answer 2

For C# to C# you can use Windows Communication Foundation. You may be able to use it with Python and C++ as well.

You may also want to checkout named pipes.

Answer 3

I would think about moving to a model where you eliminate the issue by having centralized data that all of the applications look at. Keep "one source of the truth" so to speak.

Answer 4

Most outside software has trouble linking against C++ code, due to the name-mangling algorithm it uses for its symbols. For that reason, when interfacing with programs written in other languages, it is often best to declare wrappers to things as extern "C" or inside an extern "C" { block.

Answer 5

I need to be able to transfer a data set from one application to another easily (the data set in question is not huge, probably a few megabytes)

Use the file system.

and I also need some form of way to control the current state of the operation

Again, use the file system. A "current_state.json" file with a JSON serialized object is perfect for multiple languages to work with.

It seems like sockets or maybe SOAP would be a universal solution.

Perhaps. But it's overkill for this kind of thing. Your OS already has all the facilities you need. Just use the file system. It's very simple and very reliable.

Answer 6

There are many ways to do interprocess communication. As you said, sockets may be a universal solution. SOAP, i think, is somewhat an overkill. You may also use mailslots. I wrote C++ application using it a couple of years ago. Named pipes could be also a solution, but if you are coding on Windows, it may be difficult.

In my opinion:

1. Sockets
2. Mailslots

Are the best candidates.