pub/sub events to/from microservices in different languages

Question

I'm starting a brand new project implementing microservices with domain driven design. We will have microservices written in different languages like C#, Python, and Node. I'm thinking about hosting these microservices as AWS lambda functions and using SNS and/or SQS for event pub/sub. My first attempt at hosting C# functionality as lambda functions had very poor performance when cold starting C# lambda. So, we are getting push back from business on using lambda functions.

Another option is hosting these microservices on a Linux EC2 instance. How do you implement pub/sub of events between these microservices living on a Linux EC2 instance? Can I still use AWS SNS/SQS? It's my understanding implementing a message bus across different technologies is difficult.

What's the best approach for implementing pub/sub of events across microservices built with different technologies? Thanks for your time.

Answer 1

I think your question covers too many topics and it isn't quite clear what you want to know really. I'll try anyways.

For context, the Pub/Sub solution I'm more used to is the one provided by Google Cloud Platform. I believe things will be really similar (or even better) in the AWS equivalent.

For the first part: Handling pub/sub message queues in multiple languages

As another user already said in a comment, just use a common format for this like JSON and you are golden, all languages have proper JSON readers these days. Not an issue here really

Second: The "bus" in which all tech can work with

This is really confusing to me. I'm assuming you mean the interfaces from these languages to the Message queues themselves. In that case, there should be libraries already available to your languages of choice that allows you to:

1. Pull messages from a Pub/Sub queue (and acknowledge/reject them)
2. Publish messages to a Pub/Sub queue

Worst case scenario you can probably interact with them through a REST API. I just googled a bit and found this reference for AWS SNS: https://docs.aws.amazon.com/sns/latest/api/API_Publish.html#API_Publish_Examples

I'd say it's cleaner to do things with official supported libraries (and you don't have to care about a lot of stuff), but it seems perfectly possible to handle it with whatever languages.

In your scenario with multiple languages, I wouldn't try to come up with a universal solution. Rather I'd use the best tool for each language, this way it would still feel "natural" from that language perspective and would be much easier to implement.

TL;DR

1. Use JSON for the messages
2. Use official libraries in your languages for interaction with the Pub/Sub queues
3. If not available, use the REST API to interact with the message queues