Consuming external APIs within the microservice architecture

Question

I'm writing a small microservice-based application. One of the services is assigned the task of querying an external API and processing the JSON response (filtered list of houses). Since I'm using Protocol Buffers to serialize the message that will be sent to the message broker I need to convert the JSON output to the appropriate Protocol Buffer format.

This is a diagram that illustrates the overall architecture:

The problem is, that since the JSON response is long, paginated and also has many nested fields there is no easy way to manually create a corresponding ProtoBuff message struct.

I could potentially send the JSON response as a text string and deal with it on the database service side, but this then presents the problem of how to store each house object inside the database so that it can be queried later.

Unmarshalling the whole response into a Go struct would pose the same problems as trying to create a ProtoBuff message struct in the first place. Storing it inside a database as a raw JSON field is not an option since I need to be able to query some of the fields and actually extract each house description from the body of the response.

Additionally, creating a database schema that matches hundreds of different fields and nested objects, seems like a very tedious way of going about it.

What are the best practices when it comes to processing substantial JSON output across many web services?

Answer 1

1. There is no clear reason on why to convert the JSON format in the message broker. Probably, leaving it a JSON object through the message broker is the way to go.

2. In terms of how deep you should model it, it depends on several factors. Here are some hints:

   • If it is a set of information the microservice should provide in different use cases and information other than just the external API's should be stored together, it is a sign that you should probably build your our domain or canonical model. Be careful not to grow this model too much - it should still be a microservice.
   • On the other hand, if it is basically a "cache" for the external API, storing it without any conversions should work well enough.
   • And there is always the middle path. In this case to structure part of that data, and leave part of the data unstructured.

3. The storing decision also depends on multiple factors (check the following articles and questions about Relational vs. non-relational 1, 2 and 3 - and there is much more available). You should have in mind that relationship databases (such as MySQL) are used to store relations.

   • Is it relations you are trying to store (multiple or complex relations between entities/classes)? If that is the case: tables, relations and joins should be designed. Be aware that performance could be an issue if too many relations and joins are needed.
   • Or, if every object is self contained (basically composed by aggregations), there is the possibility of storing it as an object. MySQL has some JSON features that would still allow you not just to query but to do it fast with indexes.
   • Again, you should consider if structuring only part of the data would be interesting.

Answer 2

I think your question is not particular in micro-services. It's about how to consume the external model. Generally, it's not a good idea to just consume and persist the external model directly. From domain driven design point view, the external models belong to another Domain Context. Usually, it's beneficial to have an Anti-Corruption Layer to map the external model to your own model. By this, you can de-couple your model from the external one.

This anti-Corruption Layer got pattern name "Ports & Adapters".

When you have your own model, you should align this model with the rest in your system.