Logging 3000 events per second from c program

Question

What are my best options for logging 3k events per second from a c file ? Following of the options which come to my mind. Not able to decide which would be robust solution with less failure points, higher reliability and less latency.

1. Use a messaging server to relay events as they happen

2. Use syslog for logging

3. Use Unix pipe

4. Use of logging agents like fluent which will send events to analysis server

5. Write a log file locally and then rotate periodically rotate it to analysis server using something like rsync

Answer 1

Try syslog. No reason to make it too complicated. With syslog-ng you can do local logging through UDP, then set up the local syslogd to forward everything through TCP to a central syslog server. You might need to run without fsync on the central syslog server to keep up with that load (but test first), but that can be mitigated with forwarding everything to two separate machines. This gives you the asynchronous performance locally and enough reliability that you should almost never lose events.

Another option I've done is to log events into Redis, Riak or some other nosql data store (I usually don't recommend them for anything complex, but event logging is right up their alley). Set up mirroring for redundancy and they should be able to keep up way more than 3k events per second.