How to correctly set timestamp in logstach from iis log

Question

I am trying to parse iis log files using logstash and send them to elasticsearch.

I have the following log line

2014-02-25 07:49:32 172.17.0.96 GET /config/integration - 80 - 172.17.28.37 Mozilla/5.0+(Windows+NT+6.1;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/33.0.1750.117+Safari/537.36 401 2 5 15

And use this filter:

filter {
    if [message] =~ "^#" {
    drop {}
  }

  grok {
    match => ["message", "%{TIMESTAMP_ISO8601} %{IP:host_ip} %{URIPROTO:method} %{URIPATH:path} (?:-|%{NOTSPACE:uri_query}sern) %{NUMBER:port} %{NOTSPACE:username} %{IP:client_ip} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
  }
  date {
     match => ["logtime", "YYYY-MM-dd HH:mm:ss"]  
  } 
}

Everything gets parsed correctly but in the result the @timstamp field is the time I run the parsing, not the time of the log event. This causes all the log events to end up stacked together at the time I start logstash when I view them. I would like the @timestamp to be the time of the actual event.

What am I doing wrong?

Answer 1

First, you can specific a log time field in grok. Then, you use date filter to parse the log time to @timestamp. The @timestamp will update to the log time. For example,

filter {
    if [message] =~ "^#" {
        drop {}
    }

    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:logtime} %{IP:host_ip} %{URIPROTO:method} %{URIPATH:path} (?:-|%{NOTSPACE:uri_query}sern) %{NUMBER:port} %{NOTSPACE:username} %{IP:client_ip} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
    } 
    date {
         match => ["logtime", "YYYY-MM-dd HH:mm:ss"]  
    }
}

Answer 2

I solved it, didn't realize I had to store the time from the log entry into something, in this case eventtime

grok {
    match => ["message", "%{DATESTAMP:eventtime} %{IP:host_ip} %{URIPROTO:method} %{URIPATH:path} (?:-|%{NOTSPACE:uri_query}sern) %{NUMBER:port} %{NOTSPACE:username} %{IP:client_ip} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
  } 

and then use that value to set @timestamp (wich is the implicit target field of the date filter)

date {
         match => ["eventtime", "YY-MM-dd HH:mm:ss"]  
  }

a small gotcha was the format without leading year in the date expression, I guess DATESTAMP removes the century from the year.