AWS AutoScaling, downscale - wait for processes termination

Question

I want to use AWS AutoScaling to scaledown a group of instances when SQS queue is short. These instances do some heavy work that sometimes requires 5-10 minutes to complete. And I want this work to be completed before the instance termination.

I know a lot of people should have faced the same problem. Is it possible on EC2 to handle the AWS termination request and complete all my running processes before the instance is actually terminated? What is the best approach to this?

Answer 1

Assuming you are using linux, you can create a pre-baked AMI that you use in your Launch Config attached to your Auto Scaling Group.

In the AMI you can put a script under /etc/init.d say /etc/init.d/servicesdown. This script would execute anything that you need to shutdown which would be scripts under /usr/share/services for example.

Here's kind like the gist:

servicesdown

It would always get executed when doing a graceful shutdown.

Then say on Ubuntu/Debian you would do something like this to add it to your shutdown sequence:

/usr/sbin/update-rc.d servicesdown stop 25 0 1 6 .

On CentOS/RedHat you can use the chkconfig command to add it to the right shutdown runlevel.

Answer 2

You could also use Lifecycle hooks. You would need a way to control a specific worker remotely, because AWS will select a particular instance to put in Terminating:Wait state and you need to manage that instance. You would want to take the following actions:

1. instruct the worker process running on the instance to not accept any more work.
2. wait for the worker to finish the work it already is handling
3. call the complete-lifecycle action.

AWS will take care of the rest for you.

ps. if you are using celery to power your workers then you can remotely ask a worker to shutdown gracefully. It won't shutdown unless it finishes with the tasks it had started executing.

Answer 3

I stumbled onto this problem because I didn't want to terminate an instance that was doing work. Thought I'd share my findings here. There are two ways to look at this though :

1. I need to terminate a worker, but I only want to terminate one that's not working
2. I need to terminate a SPECIFIC worker and I want that specific worker to wait until it's done with the work.

If you're goal is #1, Amazon's new "Instance Protection" looks like it was designed to resolve this.

See the below link for an example, they give this code snippet as an example: https://aws.amazon.com/blogs/aws/new-instance-protection-for-auto-scaling/

while (true)
{
  SetInstanceProtection(False);
  Work = GetNextWorkUnit();
  SetInstanceProtection(True);
  ProcessWorkUnit(Work);
  SetInstanceProtection(False);
}

I haven't tested this myself, but I see API calls related to setting the protection, so it appears that this could be integrated into the EC2 Worker App code-base and then when Scaling In, instances shouldn't be terminated if they are protected (currently working).

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/autoscaling/AmazonAutoScaling.html

Answer 4

As far as I know currently there is no option to terminate instance while gracefully shutdown and let process to complete work.

I suggest you to look at http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/as-configure-healthcheck.html.

We implemented it for resque workers while we are moving instance to unhealthy state and than downsizing AS. There is a script which checking constantly health state on each instance. Once instance moved to unhealthy state it stops all services gracefully and sending terminate signal to ec2.

Hope it helps you.