JavaMelody crashing the server with thousands of RRD files

Question

We run JavaMelody on a REST service, and have recently had trouble with the server crashing with errors like

Socket accept failed: java.net.SocketException: Too many open files

As it turns out, JavaMelody created tens of thousands of files in /tmp/javamelody. How do I fix this.

Answer 1

So it turns out that JavaMelody created unique RRD files for each URL and SQL request. Our REST service exposes hundreds of endpoints, and each of them can take thousands of values, like

http://server/get/entity/10

where 10 can be any unique ID in the system. So we had as many RRD files as there were entities being queried.

The documentation at https://github.com/javamelody/javamelody/wiki/UserGuide#6-optional-parameters hints at options called http-transform-pattern and sql-transform-pattern.

The parameter http-transform-pattern is a regular expression to transform descriptions of http requests and to delete dynamic parts (identifiers of objects for example) in order to be able to aggregate on these requests

Similarly, the parameter sql-transform-pattern is a regular expression to transform descriptions of sql requests (not binded identifiers for a "in" clause for example) in order to be able to aggregate on these requests.

That is a very vague description of what these options do, and it is not immediately obvious how they help. But others that had the same problems indicated that these settings fixed their issues, so I dug deeper.

I suspect how these options work is by replacing any part of the URL that matches the regular expression with a "$". So setting http-transform-pattern to \d+ means that the URLs http://server/get/entity/10 and http://server/get/entity/20 both have their digits matched by the regular expression, and are then aggregated into the URL http://server/get/entity/$. This in turn reduces the number of RRD files, as new ones are no longer created for every id.

In our case we used lookbehinds to group URLs. So we set http-transform-pattern to something like (?<=/server/get/fuzzballs).*|(?<=/server/get/foobars).* to aggregate the URLs into the buckets http://server/get/fuzzballs$ and /server/get/foobars$.