I'm trying to setup logstash to parse and ship logs to another server. Almost every time GROK fails to parse the logs, it causes the following error:
{:timestamp=>"2013-07-08T02:20:07.390000-0400",
:message=>"thread watchdog timeout",
:thread=>#<Thread:0x133b92c run>,
:backtrace=>["file:/opt/logstash/logstash.jar!/logstash/filterworker.rb:46:in `backtrace'",
"file:/opt/logstash/logstash.jar!/logstash/filterworker.rb:46:in `run'",
"file:/opt/logstash/logstash.jar!/logstash/agent.rb:785:in `each'",
"file:/opt/logstash/logstash.jar!/logstash/agent.rb:785:in `run_filter'",
"file:/opt/logstash/logstash.jar!/logstash/agent.rb:492:in `run_with_config'"],
:thread_watchdog=>2013-07-08 02:19:57 -0400,
:age=>10.006,
:cutoff=>10,
:state=>{:event=>#<LogStash::Event:0xa02ea9 @data={"@source"=>"file://clstaging12//home/xyz/xyz.com/apps/logs/mailerjob_log_2013_07_04.txt",
"@tags"=>[],
"@fields"=>{},
"@timestamp"=>"2013-07-08T06:19:50.114Z",
"@source_host"=>"clstaging12",
"@source_path"=>"//home/xyz/xyz.com/apps/logs/mailerjob_log_2013_07_04.txt",
"@message"=>"PID:31730 2013-07-04T13:59:03-05:00 DEBUG :[property_listing_contact.php] Inside getBrokersContactInfoReceivedUserCount of the PropertyListing",
"@type"=>"zend_log"},
@cancelled=false>,
:filter=><LogStash::Filters::Grok type=>"zend_log",
patterns_dir=>["/home/xyz/xyz.com/conf/patterns"],
pattern=>["%{ZEND_LOG}"],
match=>{"@message"=>["%{ZEND_LOG}"]},
tag_on_failure=>["_grokparsefailure"]>},
:level=>:fatal}
The pattern ZEND_LOG is :
ZEND_LOG_SIGNATURE (?:IP\:(?<clientip>(?:\%ip\%|%{IP})) )?PID\:%{NUMBER:pid}
ZEND_CONTEXT [^\]]+
ZEND_LOG %{ZEND_LOG_SIGNATURE} %{TIMESTAMP_ISO8601:timestamp} %{WORD:level} \:\[%{ZEND_CONTEXT:context}\] %{GREEDYDATA:message}
This causes the logstash agent to crash every few minutes, making it almost unusable.
I have looked at many existing bugs filed on JIRA without any luck. Here are a few links:
https://logstash.jira.com/browse/LOGSTASH-508
https://logstash.jira.com/browse/LOGSTASH-525
https://logstash.jira.com/browse/LOGSTASH-765