Amazon EC2 autoscaling down with graceful shutdown?

Question

We're looking at using EC2 autoscaling to deal with spikes in load. In our case we want to scale up instances based on an SQS queue size and then down scale with the queue size gets back under control. Each SQS message defines a potentially long running job (sometimes up to 20 minutes each for message) that must complete before the instance can be terminated.

Our software handles the shutdown process gracefully, so issuing sudo service ourapp stop will wait for the app to complete before returning.

My question; when autoscaling starts scaling down it issues a terminate (which apparently is like hitting the power button), will it wait for for our app to completely exit before the instance is 'powered off'?

https://forums.aws.amazon.com/message.jspa?messageID=180674 <- that and other things I've found seem to suggest that it doesn't

Answer 1

In mid 2014 AWS introduced 'lifecycle hooks' which allows for full control of the termination process.

Our high level down scale process is:

• Auto Scaling sends a message to a SQS queue with an instance ID
• Controller app picks up the message
• Controller app issues a 'stop instance' request
• Controller app re-queues the SQS message while the instance is stopping
• Controller app picks up the message again, checks if the instance has stopped (or re-queues the message to try again later)
• Controller app notifies Auto Scaling to 'PROCEED' with the termination
• Controller app deletes the message from the SQS queue

More details: http://docs.aws.amazon.com/autoscaling/latest/userguide/lifecycle-hooks.html

Answer 2

On most newer AMI's, the machines are given the equivalent to a 'halt' (or 'shutdown -h now' command so that the services are gracefully shut down. As long as your program plays nicely with the startup/shutdown scripts, you should be fine -- but, if your program takes more than 20 seconds to terminate, you may experience that amazon will kill the instance completely.

Amazon's documentation with regards to their autoscaling doesn't specify the termination process, but, AWS's documentation for ec2 in general does contain about what happens during the termination process -- that the machines is given a 'shutdown' command, and the default shutdown time on most systems is 30 seconds.

Answer 3

use replaceunhealty option in autoscaling.

refer: http://alestic.com/2011/11/ec2-schedule-instance

particularly see this comment.