Separate database to store logs and exceptions

Question

In an enterprise project, I need to log high volume of data (original values, modified values, user who has changed data, etc.) I'm using NLog to log such data and ELMAH to log exceptions. I'm not sure whether to use a separate database to store logging information or use the primary database. Advantages of using a separate database are:

• Logs and exceptions are not a part of business, so they don't need to be in the primary database.
• Load on the main database would reduce, resulting in a better performance.

Disadvantages are:

• If the log database stops, we will miss logs which are very important in this project. (though we can prevent saving if logging was not successful)
• Handling transactions will be hard. (What if log saved, but the data are not)

There are more advantages and disadvantages of course, but all in all, I need to know: What is the best practice in an enterprise software utilizing sizeable transactional RDBMS? Or maybe it varies from project to project based on some factors?

Answer 1

You want to separate and decouple the error logging from the data access. If data access rolls back, you want to keep the log access and log the fact that it rolled back in addition. If the database is down, you do want to log that the database is down. If your logging would be transactionally consistent with the data access then your system may experience 10 rollbacks a minute yet your error log would show none!

Another reason why logging should be its own storage (database) is that more often than not the relation is not 1:1, error logging often aggregates error reported from several systems, possible backed by several databases.

The infrastructure you use to submit error reports should handle unavailability of the error log repository (ie. logging database is down or network is partitioned), use one of the many queuing solutions available (MSMQ, RabbitMQ, ZeroMQ etc).