Logstash not parsing multiple named capture groups

Question

I have just started playing around with Logstash, ElasticSearch and Kibana for visualisation of logs and am currently experiencing some problems.

I have a log file that is being gathered by logstash and I want to extract fields from log entries before writing these into ElasticSearch.

I have define a filter with my a number of named capture groups in my logstash config file but at this point only the first of those named capture groups is matching.

My log file looks something like the following:

[2014-01-31 12:00:00] [FIELD1:SOMEVALUE] [FIELD2:SOMEVALUE]

and my logstash filter looks like the follwing:

if[type] == "mytype {    grok    {    match => [ "message", "(?<TIMESTAMP>regex)", "message", "(?<FIELD1>regex)", "message", "(?<FIELD2>regex)" ]    }    }

I have verfied the regexes for all my fields are correct but when I go to the Kibana dashboard FIELD1 and FIELD2 are not appearing.

If anyone could shed some light on this I would be grateful.

Thanks

Kevin

Answer 1

After learning a bit more about parsing using grok I've found a lot of the time it isn't necessary to have to write my own regexes. There are a number of predefined grok patterns I can use and I can extend these to create my own custom patterns when parsing logstash logs.

A useful link on the grok patterns supported by logstash: https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns.

Using the new found knowledge I was able to change my match configuration to that below.

if[type] == "mytype" {
    grok {
        match => ["\[%{TIMESTAMP_ISO8601:dateTime}\]%{SPACE}\[%{WORD}\:%{FLOATINGPOINT:cpu}\]%{SPACE}\[%{WORD}\:%{FLOATINGPOINT:memory}\]"]
    }
}

This uses the built in grok patterns TIMESTAMP:ISO8601 to pick out the date in my logs, and I have created a very simple custom pattern FLOATINGPOINT to pick out the floating point values for memory and cpu in my example. The FLOATINGPOINT pattern looks like:

FLOATINGPOINT  %{INT}\.%{INT}

Answer 2

grok's default behavior is to stop processing after the first match.

You can change this by setting break_on_match to false:

if[type] == "mytype {
    grok
    {
        match => [ 
            "message", "(?<TIMESTAMP>regex)",
            "message", "(?<FIELD1>regex)",
            "message", "(?<FIELD2>regex)"
        ]
        break_on_match => false
    }    
}