Logging architecture in microservices

Question

We are going to develop an enterprise level application which will be run by a dozen microservices. Every microservice will be inside docker and all this will be orchestrated by Kubernetes. Obviously, we need some logging solution and here is a catch. The application will have two kinds of logs, the first one is regular logs to debug application errors and see what happened if an error was thrown or something went wrong, let say technical level logs. The second one is user action logs, to understand which actions the user took and this log is supposed to be used by a user manager, not IT personnel, let's say business level logs.

I'm looking into ELK Stack solution to manage all logs and separate between technician/business logs using Elastic search filter by log's severity for example. Let's say all logs with severity level INFO will be considered as business logs, and not INFO as technical logs. It's possible that we will implement some internal format for log messages, it depends on data which technical/business log will need to store, and this format will contain a field to tell whether it's business or technical logs.

I'm looking to get some critique about my solution. Regarding the amount of data is going to be stored in logs. It's an Enterprise application with less than a hundred users per installation, so I think it will be a few gigabytes in a year, upper bound.

Answer 1

I have a few recommendations based on my experience.

1. I would call user action logs user audit logs to differentiate them from application logs.
2. ELK should be a fine place to store both kinds of logs.
3. I would use a separate index for application logs and user audit logs so you can use all logging levels for each. INFO logs can be very helpful for application logs, and you don't want to hamper your ability to use an appropriate logging level.
4. Include correlation identifiers in your logs so you can match user and application logs in troubleshooting issues.