logstash fails to match a grok filter

Question

I'm stuck. I cannot get why grok fails to match a simple regex under logstash. grok works just fine as a standalone thing. The only pattern which works for me is ".*" everything else just fails.

$ cat ./sample2-logstash.conf
input {
    stdin {}
}

filter {
    grok {
        match => [ "message1", "foo.*" ]
        add_tag => [ "this_is_foo" ]
        tag_on_failure => [ "STUPID_LOGSTASH" ]
    }
}

output {
    stdout { codec => json_lines }
}

Here's the output:

$ echo "foo" |~/bin/logstash-1.4.0/bin/logstash -f ./sample2-logstash.conf 
{"message":"foo","@version":"1","@timestamp":"2014-05-07T00:32:49.915Z","host":"serega-sv","tags":["STUPID_LOGSTASH"]}

Looks like I missed to do something in logstash because vanilla grok works just fine:

$ cat grok.conf
program {
    file "./sample.log"
    match {
        pattern: "foo.*"
        reaction: "LINE MATCHED! %{@LINE}"
    }
}

Plain grok's output:

$ echo "foo" > ./sample.log; grok -f grok.conf 
LINE MATCHED! foo

Thanks!

Answer 1

You configuration have error. The grok match field is message, instead of message1.

Then, at logstash grok page there is an example to show how to use grok. I think you have misunderstand. For example, if your log is

55.3.244.1 GET /index.html 15824 0.043

The grok pattern for logstash is

%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}

For %{IP:client}, The first parameter (IP) is grok pattern, the second parameter(client) is the field you want to put this message.

Answer 2

Everything @Ben Lim said. The very next section of the documentation shows how to apply semantics to generic regex syntax:

filter {
  grok {
    match => [ "message", 
      "^(?<ip>\S+) (?<verb>\S+) (?<request>\S+) (?<bytes>\S+) (?<delay>\S+)$"
    ] 
  }
}