Centralized Logging - Correlating Messages Across Servers

Question

We have a very distributed system. A user's request on the site may involve calls to several services. For example, when a user log onto the site, calls are made to ads service, personalization service, related news service, etc. to construct the data needed for display up on login. High-level design: A request to a URL is mapped to a Spring MVC Controller and that controller makes the call (most of them using HttpClient) to different services.

We are implementing a centralized log solution using Logstash, ElasticSearch, Kibana, Log4j/SLF4J. When a issue is reported on the site, we want to be able to change log level to debug and see the log messages for a particular request across all services. We are populating request id in Log4j MDC, so we are able to identify log messages for that particular request on the webapp server. How do I go about correlating messages from the calls made to other services?

Flow: User log in --> request mapped to Spring MVC Controller which logs messages by populating request id in Log4j MDC --> http client calls to service1, service2, service 3

How to correlate messages from service1, service2, service3 to the messages logged by MVC controller. One solution is to pass the request id in http client calls. There are lot of applications that follow this paradigm so changing code everywhere is not an ideal solution.

UPDATE1:

I don't know much about jvm agents, but I'm wondering if a custom agent can be developed to intercept network calls and add a parameter. The custome agent on the receiving side will detect the parameter add it to a ThreadLocal variable. Dynatrace PurePath technology somehow correlates calls across JVMs - they require injecting their jvm agent, so I'm guessing they are intercepting calls in the agent. Check out this video

Answer 1

You're going to have to bite the bullet and add the request ID to the HTTP client calls. If you don't want to pollute your APIs, add it as a custom HTTP header, then extract it using some kind of HTTP interceptor on the service side (depends on what web service stack you're using), and re-add it to the MDC.