How to make an EC2 instance automatically reboot when ELB says it's unavailable?

Question

Every now and then Elastic Load Balancer (ELB) kicks out one of my servers for being unavailable. Which is good -- hardly any interruption.

Usually, just rebooting the instance through the AWS Console fixes the problem.

I would like to have my EC2 instance automatically reboot when it becomes unavailable.

What's the best way to do this? (it happens to be a Windows instance, but ideally that shouldn't matter)

Answer 1

I got a cheap EC2 micro instance and wrote a simple script that lists all my servers on EC2, and does a GET on each one to make sure that it's functioning properly. If a server isn't functioning properly, my script reboots it. I have the script running once every 10 minutes.

Woulda been slightly easier if there were a service out there to do this for me, but the system was easy to set up and the micro instance is cheap.

Answer 2

Use autoscaling group and set it to keep 1 live instance. If your instance crashes it will spawn a new one from predefined instance image. It's easy and convenient.

Answer 3

Agree with @agmin that a lambda function is a great way to do this. You can configure a CloudWatch alarm to trigger the lambda function via an SNS topic.

I put this implementation together and it seems to be working well. It can be set up and configured entirely via the command line:

https://github.com/jcowley/elb-instance-reboot

Answer 4

Generally speaking you should have your load balancers set up with an auto scaling group. The auto scaling group will have information about how your server is configured. For example it will have an ami and possible user-data-file that will allow it to be configured without any user interaction. When you have these two services working together the load balancer will terminate your failed instance and start new instance immediately.

Seeing as your using a windows instance you could write a powershell script to configure your server and use CloudInit.NET to execute the script when the server starts. The last part of the script should enable the check on the load balancer.

Answer 5

Write a lambda function!

It's cheaper than running a micro instance, and you can set it to run at whatever interval you'd like. You also get CloudWatch monitoring with it so you don't need to worry about checking the health of your micro instance separately.

I'm using a ElasticBeanstalk, which offers the describeInstancesHealth endpoint. Alternatively you can use ec2's describeInstanceStatus endpoint.

Then if you detect a bad instance, ec2's rebootInstances let's you reboot one or more instances.