Is the @timestamp field needed when using Logstash to store in Elasticsearch?

Question

I have the following setup: I have a Java tool which sends JSON messages to RabbitMQ. They look like this:

{
"a": 0,
"b": 1,
"c": 2
}

Now I use Logstash to read the RabbitMQ queue and store them into Elasticsearch, so I can analyze the data with Kibana. The JSON stored in Elasticsearch looks like this:

{
"a": 0,
"b": 1,
"c": 2,
"@version": "1",
"@timestamp": "2014-01-22T19:05:19.136Z"
}

I don't think the @timestamp field will be of any use for what I'm doing. When I use cURL to store the same JSON in Elasticsearch, only the @version field is there, the @timestamp field is not present. Is there any way to configure Logstash to not save @timestamp?

Answer 1

When a message is read by Logstash, Logstash treat the message as a Event. An event will have a timestamp and message log. Thus, the @timestamp field is requisite.

Therefore, if you want to delete the @timestamp field, it will causes an error. Logstash can't output the event to the elasticsearch.

Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `tv_sec' for nil:NilClass
    at RUBY.sprintf(file:/tmp/logstash-1.2.1-flatjar.jar!/logstash/event.rb:239)
    at org.jruby.RubyString.gsub(org/jruby/RubyString.java:3062)
    at RUBY.sprintf(file:/tmp/logstash-1.2.1-flatjar.jar!/logstash/event.rb:225)
    at RUBY.receive(file:/tmp/logstash-1.2.1-flatjar.jar!/logstash/outputs/elasticsearch.rb:153)

So far, not all @-prefix fields causes error, only remove @timestamp will cause this error.

Answer 2

Any @-prefixed field is used internally by Logstash. Removing them tends to cause errors.

For example, I tried the following config file with Logstash 1.3.3:

input { 
    generator {
        type => "timestrip"
        message => "This is a test message."
        count => 1
    }
}

filter {
    mutate {
        remove_field => ["@timestamp"]
    }
}

output {
    elasticsearch_http {
        host => "127.0.0.1"
        flush_size => 1
    }
}

The ES output reports "Failed to flush outgoing items" with the following exception:

NoMethodError: undefined method `tv_sec' for nil:NilClass