Gearmand Server Completely Locks Up

Question

Running Gearmand 1.1.8 on a CentOS AWS VM using MySQL as storage for the queue, every few hours Gearmand suddenly spins out of control, 100% CPU, and sucks up most of the memory on the small instance.

We are currently in testing and not production, all the messages we send to it are relatively small and well formed, the biggest of which is aprox 15 megabytes.

We initially hit a wall with the bigger requests but increased mysql's max packet size and InnoDB's max log size because it was barking about both

[mysqld]
max_allowed_packet=20M
innodb_log_file_size=300M 

We are using the recommend schema http://gearman.info/gearmand/queues/mysql.html in InnoDB (the schema doesn't specify storage)

We have 20 workers connected to the gearmand server, and it ran FINE stacking up hundreds of thousands of messages until the last few days.

Any help would be greatly appreciated.

While this is going on there is no network traffic in or out. We have a status dashboard (see: http://jdon.at/kmCU) which completely locks up waiting for a STATUS response from the Gearman server.

Any help would be greatly appreciated! Thanks

Answer 1

The problem was a but in the version we were using, 1.1.8 whereas the gearmand queue server would crash upon receiving a WORK_EXCEPTION Upgrading to 1.1.11 corrected the issue.

see: https://bugs.launchpad.net/gearmand/+bug/1164997