Logstash - grok not parsing even with grokdebugger being happy

Question

Im at a loss and probably should step away from the problem, Can anyone help spot what I am missing. Logstash keep thowing "_grokparsefailure". Scratching my head???

using logstash logstash-1.3.3-flatjar.jar

Log file example

proxy.ian.com - ian@IAN.COM [24/Feb/2014:11:16:49 -0500] "GET /docs/en-US/Guide/+ HTTP/1.1" 404 285 "https://ian.com/docs/en-US/Guides/html/Guide" "Mozilla/5.0 (X11; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0"

My logstash filter

Filter {
   if [type] == "ErcAccess" {
    grok {
    match => ["message", "%{IPORHOST:clientip} - %{USER:auth}@%{URIPROTO}.%{WORD:domain} \[%{HTTPDATE:timestamp}\] "%{WORD:httpmethod} %{NOTSPACE:referrer} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} "%{NOTSPACE:request}" %{QS:UserAgent}" ]
         }
     }
}

Answer 1

Your pattern includes " characters, which are apparently treated as literal characters by grokdebug. When Logstash is reading your config file, those quote characters have a different semantic meaning (they mark the beginning or ending of a string).

UPDATE: turns out Logstash's escaping of quotes is poorly documented and possibly buggy. I'll update if I find a better solution, but for now it looks like you can use ' single quotes to begin/end your strings, which will allow you to use " double quotes freely within them.

This works for me:

input {
    generator {
        type => 'ErcAccess'
        message => 'proxy.ian.com - ian@IAN.COM [24/Feb/2014:11:16:49 -0500] "GET /docs/en-US/Guide/+ HTTP/1.1" 404 285 "https://ian.com/docs/en-US/Guides/html/Guide" "Mozilla/5.0 (X11; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0"'
        count => 1
    }
}

filter {
  if [type] == 'ErcAccess' {
    grok {
      match => ['message', '%{IPORHOST:clientip} - %{USER:auth}@%{URIPROTO}.%{WORD:domain} \[%{HTTPDATE:timestamp}\] "%{WORD:httpmethod} %{NOTSPACE:referrer} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} "%{NOTSPACE:request}" %{QS:UserAgent}' ]
    }
  }
}

output {
    stdout {
        codec => rubydebug{}
    }
}