Why is Logstash importing the same file every time I restart it?

Question

Why does logstash import the same file everytime I restart it?

I have 3 different log files within the same folder (C:/temp/ in this example). Everytime I restart logstash, the files are being imported again! Is this expected?

logstash.conf

input { 
  file { 
    type => "app" 
    path => [ "C:/temp/*.log" ] 
    codec => "json" 
  }
}

output { 
  stdout { codec => rubydebug }
  elasticsearch { embedded => true }
}

This is the command which I use to run logstash (I'm using Windows):

java -jar logstash-1.3.3-flatjar.jar agent -f logstash.conf -- web

I know the files are being imported everytime I restart because they are being output to the console and I can see the same log messages appearing several times when browsing the logs through kibana.

Answer 1

Answering my own question:

In fact, this is not expected and it's a known bug for windows. You can check it here here and here (quote below).

After several tests on Linux and Windows, I realized that if logstash is (re-) started on Windows, all logs are parsed from the beginning again. It does not matter if I set a start_position or an since_db path, or an combination of both, the setting will be ignored.

UPDATE:

I fixed this by manually patching logstash-1.3.3-flatjar.jar with some changes made by edwinf to ruby-filewatch. In case you want to do the same:

1. Open https://github.com/jordansissel/ruby-filewatch/tree/master/lib/filewatch and download buftok.rb, tail.rb, watch.rb and winhelper.rb
2. Open logstash-1.3.3-flatjar.jar using any zip file editor and put the downloaded files inside the folder named "filewatch" (replace the original files)
3. Download JRubyFileExtension.jar from https://github.com/jordansissel/ruby-filewatch/tree/master/java
4. Put this file at the root of logstash-1.3.3-flatjar.jar (while opening it with a zip file editor).
5. Done. Logstash must be rerun, of course :)

The pull request which fixes this issue can be found at https://github.com/jordansissel/ruby-filewatch/pull/22