Get rsyslog forwarding messages after remote server restart

Question

I have syslog successfully forwarding logs to an upstream server like so:

$MainMsgQueyeType LinkedList
$MainMsgQueueSize 10000
$MainMsgQueusDiscardMark 8000
$MainMsgQueueDiscardSeverity 1
$MainMsgQueueSaveOnShutdown off
$MainMsgQueueTimeoutEnqueue 0

$ActionQueueType LinkedList    # in memory queue
$ActionQueueFileName fwdRule1  # unique name prefix for spool files
$ActionQueueSize 10000         # Only allow 10000 elements in the queue
$ActionQueueDiscardMark 8000   # Only allow 8000 elements in the queue before dropping msgs
$ActionQueueDiscardSeverity 1  # Discard Alert,Critical,Error,Warning,Notice,Info,Debug, NOT Emergency
$ActionQueueSaveOnShutdown off # save messages to disk on shutdown
$ActionQueueTimeoutEnqueue 0
$ActionResumeRetryCount -1     # infinite retries if host is down

$RepeatedMsgReduction off

*.* @@remoteserver.mynetwork.com:5544

On the remoteserver I have something that talks syslog and listens on that port. To test, I have a simple log client that logs 100 messages a second to syslog.

This all works fine, and I have configured the queues above so that in the event that the remoteserver is unavailable, the queues start filling up, and then eventually messages get discarded, thus safeguarding syslog from blocking its logging clients.

When I stop the remote log sink on remoteserver:5544, syslog is still stable (queues filling up / full up), but when I restart the remote log sink a while later, rsyslog detects the server again, reestablishes a TCP connection

HOWEVER - syslog only forwards 1 message to it, despite the queue having many thousands of messages in it, and the logging client continuing to log 100 messages a second

How can I make syslog start forwarding messages again once it has detected the remoteserver is back up? (Without restarting syslog).

Am using rsyslog 4.6.2-2

I am using, and want to use TCP

Answer 1

The problem in case anybody comes across this was that workdirectory was set to:

$WorkDirectory /var/spool/rsyslog  

And the above config, does this:

$ActionQueueFileName fwdRule1

Even though its supposed to be an in-memory queue. Because of this, when the queue reached 800 (bizarrely, not 8000), disk-assisted mode was activated, and syslog attempted to write messages to /var/spool/rsyslog. This directory didn't exist . Randomly, (hence a race condition must exist and a bug in rsyslog), after continually trying to open a queue file on the disk in that directory, rsyslog got into a twisted state and gave up and continued queueing messages, until it hit the high 10,000 mark. Restarting the downstream logserver failed to make it recover.

Taking out all references to ActionQueueFileName and making WorkDirectory exist fixed this issue.