Centralized Logging for many Java Apps: Syslog vs JMS vs Http vs Local file

Question

I want all my applications log to be centralized (ideally in near real-time). We will use a Log4 Appender. Which one should I use:

1. Send log event in a JMS Queue
2. Syslog / syslog-ng
3. Write to a localfile and use rsync (every 3second) to replicate the log
4. Do a POST to a centralized REST Http Service

Which one are you using?

Answer 1

Depends on your requirements. Logging to a queue (as in JMS) gives you the most flexibility of your options as your log operation can return as soon as the log message has been written to the queue. You are then free to have another process take those log messages off the queue and write them to your preferred logging store (database, file system, ...).

The (slight) downside is your log messages will lag slightly behind the system that is logging, but this is almost always the case, even with file-based logging.

Answer 2

This may be overkill but have you looked at Splunk? For Java apps, I've used either Log4J or SLF4J to log to files, then Splunk can aggregate the logs and make them searchable.

Answer 3

I'm not familiar with Log4 (Log4j?), but does it have a database appender? This would provide the centralization you need and would eliminate any need to replicate your logs.

Answer 4

It really very much depends on what you want.. I don't think syslog is a good choice for application logging though, it's more suitable for infrastructure logs. Without knowing too much details, all 4 options you listed look OK. We use this.

Answer 5

btw, log4j does have AsyncAppender which uses queue and dedicated consumer-thread for logging from it