Design and setup a centralized logging with microservices / docker

Question

I am developing on a a large webapp system that has multiple docker containers. Imagine a webapp but with many external restful services running on different containers. A microservices setup.

I am looking to put in place a centralized logging system. So we can trace and find errors.

My initial thoughts was to spin up a elk (Elastic Stack) container and each of the containers logs directly to the central place.

Something like the following log config in each of the containers that want to log to the centralised elk container

<appenders>
    <Socket name="logstash" host="docker-url" port="7007">
        <SerializedLayout/>
    </Socket>
    <async name="async" blocking="false">
        <appenderref ref="logstash" />
    </async>
.... 

So my questions is this.
Is this a good design, could this be improved? I'm concerned that if the logging elk container went down then it may make the other containers unresponsive.

Answer 1

I would say this is a typical way to implement centralized logging.

Is this a good design, could this be improved?

In my opinion it's just fine

I'm concerned that if the logging elk container went down then it may make the other containers unresponsive.

Your logging framework should not affect the performance of your application if the logging appender is unavailable. It looks like you've specified that logging should be non-blocking and asynchronous. This should be enough. SLF4J implementation Logback claims to be:

Graceful recovery from I/O failures

Logback's FileAppender and all its sub-classes, including RollingFileAppender, can gracefully recover from I/O failures. Thus, if a file server fails temporarily, you no longer need to restart your application just to get logging working again. As soon as the file server comes back up, the relevant logback appender will transparently and quickly recover from the previous error condition.

It would be easy for you to test what happens when the ELK container goes down.