How can I auto scale EC2 instances with system data collected by the Sensu framework?

Question

I want to scale out my EC2 instances on AWS. For this I have been suggested to use the Sensu framwork.

I want to scale out the instance based on its CPU usage. For testing I have configured Sensu on both Windows and Ubuntu(V.Box), I'm running a client on Ubuntu by following this example. My CPU data is successfully passed to RabbitMQ.

Now I'm wondering how I can use that data in the Sensu server so that I can scale in or scale out? Any suggestion will be appreciated.

In case it matters, I will use this with Opscode Chef.

Answer 1

The easiest way to achieve your goal would be to connect the available components together (which will still require writing some code, see below) and refrain from adding custom solutions as much as possible:

Amazon EC2 offers Auto Scaling, which is in turn be driven by Metrics collected via Amazon CloudWatch. So metrics are key here, and that's exactly what Sensu is all about, see e.g. Sensu and Graphite, which covers two approaches for pushing metrics from Sensu to Graphite:

Remember: think of Sensu as the "monitoring router". While we are going to show how to push metrics to Graphite, it is just as easy to push metrics to any other system – Librato, Cube, OpenTSDB, etc. In fact, it would not be difficult at all to push metrics to multiple graphing backends in a fanout manner. [emphasis mine]

Your metrics are available in the Sensu server already, so you'll need to push them into CloudWatch now (just like explained for Graphite in the article above) and attach respective Auto Scaling policies to these in turn.

The currently available metrics handlers for Sensu are targeting Graphite and Librato indeed, so you'd need to implement such a Sensu Handler for Publishing Custom Metrics into CloudWatch (be sure to share it, it will definitely be widely used over time :)

Good luck!