Cleanup Chef node/client on Cloudformation stack deletion

Question

I have a cloudformation stack that consists of a VPC, two subnets (public and private), several EC2 ubuntu instances and all of the routes, EIP addresses, etc. One of the EC2 instances is in a public subnet. It is bootstrapped as a Chef node on startup.

I'd like to figure out a way to delete the chef node when the cloudformation stack is deleted. So far I've tried dropping a cleanup script into EC2 instance /etc/rc0.d.

This script does what it should when run manually, however when I just delete the stack, it does not seem to run. Actually - it might very well run, but I'm guessing that by the time the EC2 instance shuts down all of the routing and EIP addresses might already be gone, so Chef server might not be reachable by the EC2 instance.

I've also tried locking down creation/deletion order with 'DependsOn' attributes, but that didn't work out either - I don't think it's possible to have the IP and routes depend on the instance that is using the said EIP and routes

Is there some way to setup some sort of monitoring that will make sure Chef cleanup runs before everything else?

Gist with the template and chef setup/cleanup script

Answer 1

Yes, most likely your IPs are disassociated/removed before the instance shuts down, making any attempt to reach the Chef server from the instance futile. You can always check your cloudformation action logs but disassociating the IP address before shutdown is what makes most sense.

I think some of the workaround are:

1. Build an app on top of your cloudformation creation so that every time you delete a stack it also deletes the node(s) you want from your chef server. This would a full blown application with a database to keep track of the servers/stacks running. This will require your app to call the chef server API or simply call a system knife command.

2. Run your clean up script from another instance running knife/chef-client. You can have some sort of cron/periodic job checking for stacks/servers that have been deleting on AWS and then run the appropriate knife command to delete the server from. This in essence very similar to 1. with just the difference that you don't necessarily have to build a full blown.application.

Hope it helps.