Setting a trace id in nginx load balancer

Question

I'm using nginx as a load balancer in front of several upstream app servers and I want to set a trace id to use to correlate requests with the app server logs. What's the best way to do that in Nginx, is there a good 3rd party module for this?

Otherwise a pretty simple way would be to base it off of timestamp (possibly plus a random number if that's not precise enough) and set it as an extra header on the request, but the only set_header command I see in the docs is for setting a response header.

Answer 1

In most cases you don't need a custom module, you can simply set a header with a combination of embedded variables of http_core_module which is (most probably) unique. Example:

  location / {
      proxy_pass http://upstream;
      proxy_set_header X-Request-Id $pid-$msec-$remote_addr-$request_length;
  }

This would yield a request id like "31725-1406109429.299-127.0.0.1-1227" and should be "unique enough" to serve as a trace id.

Answer 2

nginx 1.11.0 added the new variable $request_id which is a unique identifier, so you can do something like:

   location / {
      proxy_pass http://upstream;
      proxy_set_header X-Request-Id $request_id;
   }

See reference at http://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_id

Answer 3

Old question, new answer suitable for nginx verions 1.3.8, 1.2.5 and above.

You can use a combination of $connection and $connection_requests now. Just define your own variable in the server block:

server {
    ...
    set  $trace_id  $connection-$connection_requests;
    ...
}

This id is going to be unique across nginx unless the server gets restarted.

$connection - The connection serial number. This is a unique number assigned by nginx to each connection. If multiple requests are received on a single connection, they will all have the same connection serial number. Serial numbers reset when the master nginx process is terminated, so they will not be unique over long periods of time.

$connection_requests - The number of requests made through this $connection.

Then, in your location block, set the actual trace ID:

location / {
    ...
    proxy_set_header X-Request-Id $trace_id;
    ...
}

Bonus: Make $trace_id unique even after server restarts:

set  $trace_id  $connection-$connection_requests-$msec;

$msec - The current unix timestamp in seconds, with millisecond resolution (float).

Answer 4

In our production environment we do have a custom module like this. It can generate a unique trace id and then it will be pushed into http headers which send to the upstream server. The upstream server will check if the certain field is set, it will get the value and write it to access_log, thus, we can trace the request.

And I find an 3rd party module just looks the same: nginx-operationid, hope it is helpful.