Kill hung unicorn worker processes?

Question

I am currently running a large rails application on a virtual server. My setup uses Unicorn to serve the Rails App with nginx as the proxy server in front.

Occasionally I get notifications that My CPU usage has pegged for a long amount of time. Then I login and use htop to find the badly behaving process to kill it. Those misbehaving processes always seem to be unicorn workers.

What I am wanting to know is, if there is a way to setup monit, or god to monitor these 'child processes' of unicorn and kill them if they hang at high cpu usage for a long period of time. I know how to monitor the unicorn master process because it writes it's process id to a PID file. But it is never the master process, only the child processes the misbehave.

Answer 1

You might want to try the process outlined here: Where Unicorns go to die: Watching unicorn workers with monit

I haven't done it yet, but I'm going to give it a shot shortly.

Edit: It works really well. I'm using the methods there in production now.

Answer 2

I use God to monitor my Unicorns (That just sounds too cool). I've been using the same configuration as GitHub use outlined here. In the second Gist you'll see a custom script written to monitor unicorn workers. It's a little hacky, but it works well. I believe something like this is to be merged into God at some point. But I haven't heard anything about that in a while.