Microservices shared data

Question

I'm designing a microservices structure, and I'm facing some problems on how to place data on different microservices.

For example, I have users that subscribe to plans and the plans have different features. For example, a user can download 10 items per month.

So I'm thinking of building the following microservices:

• User microservice: Maintain users data and the downloads
• Plans microservice: Maintain plans data and the features each plan enables
• Other microservices: Other microservices that may use the previous two to check permissions So when a user requests a new download, I need to get the user's actual amount of downloads and the user plan. And then check if the actual amount of downloads allows a new download based on the plan amount of download limit. If it's allowed, then I need to reach the users microservice to update the amount of downloads.

When I'm thinking about this design, I'm not sure of the following:

Where should I store what plan a user has (Users vs Plans microservices)? Should the microservices communicate with HTTP?

Thanks

Answer 1

It's all about the data

Many prominent computer scientists can be quoted saying something like this. For example, of building Git, Linus Torvalds said:

Bad programmers worry about the code. Good programmers worry about data structures and their relationships.

If you don't have the data model figured out, you will be continuously refactoring your microservices, as you continue to change your data model.

So start with a matured data model. Review it, try to break it with edge cases; then move on to "code".

CRUD then Orchestration

CRUD services are those that are for INSERT,READ,UPDATE,DELETE. They are tightly correlated to a table/collection. These are actually the closest to the academic definition of microservices: "own data".

However, information arises from relationships between data. Orchestration services are a layer of microservices that may be designed for particular tasks. These don't need to "call" the CRUD services, particularly if you already can enforce data-access rules in the data-layer.

Where should I store what plan a user has (Users vs Plans microservices)?

This question arises when designers limit themselves to "CRUD Service" thinking. You need a orchestration services.

Subscribing to a plan, and Downloading a file, probably need to interact with a few table/collection in your database. They would be orchestration.

---

Microprocess discipline will help

Microservices are not a well-defined discipline, but Microprocess Architecture is. see https://colossal.gitbook.io/microprocess/comparisons/compared-to-microservices.

There are no CRUD microservices - your client application can directly query the database. There are no Orchestration microservices - you can model these as data.

Ideally you would conceptually model a "FileDownload" entity, PlanSubscribe, and more directly in your database model.

Model:

• Users
• DownloadsRequest - when a user selects an item to download, this record is created. The user is allowed to INSERT into this table.
• DownloadTokens - a microprocess creates this when there is a DownloadRequest record without a DownloadToken, and if the user hasn't gone over monthly limit. The user is allowed to SELECT from a view of this table where the UserID field is themself.
• DownloadLogs - when the file download occurs, the system creates this record. This is used to count how many successful downloads occur. It's possible to log Download-Start and Download-End stages.
• Plans - there are many ways that plans could be limited. This is just the placeholder for the name.
• PlanDownloadLimits - limits should be modelled in separate tables.

Then you just mutate data without worrying about "service" call structures.

Example Pseudo-code Microprocess for Download Tokens:

foreach R in (select R.*
from DownloadsRequest R
where not exists (select 1 from DownloadTokens T where T.RequestID = R.ID) -- Not yet created
and exists (select 1 from PlanDownloadLimits L where L.UserID = R.UserID and L.IsActive=1 and MonthlyLimit > 0) -- Quick check that they are on a valid plan)

#Downloads =  (select count(*) from DownloadLogs O where O.UserID = R.UserID and year(O.Started)=year(now()) and month(O.Started)=month(now()))

#Limit = (select count(*) from DownloadLogs O where O.UserID = R.UserID and year(O.Started)=year(now()) and month(O.Started)=month(now()))

if (#Downloads < #Limit)
    INSERT INTO DownloadTokens (..) values (R.UserID, R.DownloadID, R.Download.URL, SecureRandomForToken())
else
    Update DownloadsRequest set Denied=1 where ID = R.ID

If you don't want to use the "Microprocess" discipline, that's fine. But you will at least obtain a more complete conceptual model by thinking this way.

Answer 2

That doesn't feel like a good way to organise your micro services.

Might I suggest sketching out the operations first and the data they need:

• Use a hard line between an operation and its data if it's critical that it's fresh and current to the operation
• Use a dotted line between an operation and data if it's okay to be unavailable, stale, or is ancillary to the operation.
• Use a dotted line if the data is passed into the operation (such as user tokens), or is passed out of the operation (like a returned answer), also place the passed in/out data in a circle using a hard line.
• An operation can be split into two or more operations with an arrow using a dotted line. Each operation is its own fully contained useful piece of work, the dotted lines point to the operations started after this one completes, to hand the work off too.

Then draw bounding bubbles around operations and data. The bounding bubbles can only intersect dotted lines.

Those bubbles are your modules. One or more modules linked by dotted lines make a service. If two bubbles cannot be connected by dotted lines directly or by other bubbles then they are strictly separate services.

The fewer modules the more micro the service.