Converting date MMM dd HH:mm:ss for logstash

Question

I have a logfile with a custom format, the date field looks like this:

Dec  4 23:59:21
Nov 21 23:59:21

in my logstash config I have this for the filter:

date {
    type => "custom"
    # tell it the format
    custom_timestamp => ["MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
    # locale didn't help
    locale => "en"
}
mutate{
    type => "custom"
    # replace the timestamp
    replace => ["@timestamp", "%{custom_timestamp}"]
}

which supposedly replaces the logstash timestamp with the custom one from the logs (I am backfilling it from old logs at the moment to test).

If I turn on the debug flag and output to stdout, it shows me that @timestamp has been replaced with custom_timestamp but I get an error message telling me that it cannot be imported:

:exception=>java.lang.IllegalArgumentException: Invalid format: "Dec  4 23:59:21"

what do I have to do to convert the date format?

Answer 1

Turns out that the sample I was working from is wrong. You do not need the mutate replacement, the config is this now:

date {
    type => "custom"
    # tell it the format
    custom_timestamp => ["MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
    # date format is english, computer might not be
    locale => "en"
}
mutate{
    type => "custom"
    #other mutations go here
}

Answer 2

Two misconceptions in this post:

1. The java exception is generated because there is no YEAR in your format, therefore it cannot parse the date safely.
2. You need to run a mutate if you want other applications to see your old imported logs as a coherent timeline. Otherwise, when you import all your old logs, you'll only see a few minutes of events concentrated (during the import).

Other than that, good question/answer, it helped me get back on track on my particular problem ;)