splunk parsing IIS logfile

Question

I am using Splunk to parse IIS logfiles from a few servers, all the servers have same fields setup in IIS and all servers running same version of windows 2003 server. However splunk tag the sourcetype of those logfile to "iis" or "iis-2" or "iis-3"... even from same server. I don't seem to be able to find the pattern. How to make sure splunk tag all logfile same type?

another question is that for some logfiles, splunk automatically extract all the key/value in the querystring fields, while doesn't for some logfiles... I'd like to have splunk to parse out the querystring key/value at the index time so it will be quick during search time.

anyone help?

Thanks

Answer 1

IIS logs are very easy to splunk, but you need to tell it what format the logs are in (since you can alter the log format). Here is an example for you.

In inputs.conf ($SPLUNK_HOME\etc\system\local\inputs.conf), add a stanza like this:

[monitor://C:\inetpub\logs\LogFiles\W3SVC1\*.log]
sourcetype=MSWindows:2008R2:IIS
queue=parsingQueue
index=msexchange
disabled=false

In props.conf ($SPLUNK_HOME\etc\system\local\props.conf), add a stanza like this:

[MSWindows:2008R2:IIS]
TZ = GMT
SHOULD_LINEMERGE = false
CHECK_FOR_HEADER = false
REPORT-fields = mswin_2008r2_iis_fields
TRANSFORMS-comments = ignore_comments

Finally, we need to define the two transforms in transforms.conf (which is in $SPLUNK_HOME\etc\system\local\transforms.conf) as follows:

[ignore_comments]
REGEX = ^#.*
DEST_KEY = queue
FORMAT = nullQueue

[mswin_2008r2_iis_fields]
FIELDS = "date","time","s_ip","cs_method","cs_uri_stem","cs_uri_query","s_port","cs_username","c_ip","cs_user_agent","sc_status","sc_substatus","sc_win32_status","time_taken"
DELIMS = " "

The format of the mswin_2008r2_iis_fields is taken from the top of the IIS log file. This is (hopefully obviously) for the default IIS logs from Windows Server 2008 R2. The location and format have changed from version to version, plus you can alter both the location and format on a per-host basis.

For more information on these configuration files, see the documentation - freely available at http://docs.splunk.com

Answer 2

Splunk will auto-recognize many kinds of data sources, called "sourcetypes" in Splunk, and if you don't tell Splunk which specific "sourcetype" to use, it will create a new sourcetype and name it based on which sourcetype it thinks it matches.
To prevent this feature from lumping together sourcetypes that might be different, Splunk will create a new sourctype with a sequential number appended. That's why you're seeing "iis-2" "iis-3" etc. You should either choose the "iis" sourcetype when you create your new data inputs, if it matches your data, or create a new "iis-whatever" sourcetype for your data.