Aggregation of IIS logs

Question

We have an IIS .Net application deployed across several machines. We use IIS log information to do reporting of performance of the web application and navigation by the user. Currently the reporting is only required infrequently (once a day, for the previous day), so we just roll the logs every 24 hours, and move the old logs to our reporting server.

We have a new requirement that means we need much faster turnaround on the IIS log information, say every minute for the sake of the discussion.

There exist Apache tools like Facebook's Scribe to scalably move Apache web server logs across a network of servers.

Are there any similar tools available for IIS?

Is this the right question to ask?

Should we be doing something different, if the timing requirements have changed so much?

I've looked at this question and the answers, and the only one that seems to come close is this one.

Pointers appreciated!

Answer 1

Snare is a little old but worth mentioning.

Snare Agent for IIS Servers

http://www.intersectalliance.com/projects/SnareIIS/index.html

I used this old version a long time ago and it worked well by forwarding/sending/replicating IIS logs over a network via syslog.

Today, they have a newer version called Snare Epilog

http://www.intersectalliance.com/projects/EpilogWindows/index.html

The code is also open source; perhaps you might find it useful.

You might also want to try ...

http://nxlog.org

http://www.syslogserver.com/syslogagent.html

Answer 2

I tend to write a .bat file in conjunction with LOG Parser 2.2. The .Bat file will determine the appropriate file dates and pull the corresponding logs from multiple IIS server log locations into a single local directory. Once the files are across I then run a Log Parser command to query the log content over all log files and then produce a single output file in .csv format. Finally, I run an SSIS job to import the new .csv file into a running log table which I can then query on an ongoing basis.