How to speed up logstash pattern matching (grok)?

Question

I have a log file with 200 MB. I feed the log file into logstash, and it is taking few hours to get the job done.

I am wondering if there's a way to speed things up? Perhaps running it in parallel mode?

Answer 1

You can take a look at here about how to speed up.

The default number of filter workers is 1, but you can increase this number with the '-w' flag on the agent.

For example, If your grok pattern is complex, you can use multiple filter worker(thread) to do the filter task and speed up logstash parsing the logs.

Answer 2

Start with 10 workers like this:

`bin/logstash -f test.conf -w 10`

Will output

Settings: User set filter workers: 10, Default filter workers: 1
Logstash startup completed