ELB Health Check not checking web instance after booting up

Question

We have a web instance (nginx) behind a ELB which we manually power on when required. The web app starts up quickly and returns a successful 200 response when we run wget locally.

However the website will not load as the ELB isn't sending healthcheck requests to the instance. I can confirm this by viewing the nginx access logs.

The workaround I've been using is to remove the web instance from the ELB and add it back in. This seem to activate the healthchecks again and they are visible from our access logs.

I've edited our Healthcheck settings to allow a longer timeout and raise the Unhealthy Threshold to 3 but this has made no difference.

Currently our Health Check Config is: Ping Target: HTTPS:443/login Timeout: 10 sec Interval: 12 sec Unhealthy: 2 Healthy: 2

Listener: HTTPS 443 to HTTPS 443 SSL Cert

The ELB and web instance are both on the same public VPC Security Group which has http/https opened to 0.0.0.0/0

Can anyone help me figure out why the ELB Health checks aren't kicking in as soon as the web instance has started? Is this by design or is there a way of automatically initiating the checks? Thank you.

Niall

Answer 1

Does your instance come up with a different IP address each time you start it?

Elastic Load Balancing registers your load balancer with your EC2 instances using the IP addresses that are associated with your instances. When an instance is stopped and then restarted, the IP address associated with your instance changes. Your load balancer cannot recognize the new IP address, which prevents it from routing traffic to your instances.

^{— http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/TerminologyandKeyConcepts.html#registerinstance}

It would seem like the appropriate approach to getting the instance re-associated would be for code running on the web server instance to programmatically register itself with the load balancer via the API when the startup process determines that the instance is ready for web traffic.

---

Update:

Luke@AWS: "You should be de-registering from your ELB during a stop/start."

— https://forums.aws.amazon.com/thread.jspa?messageID=463835

I'm curious what the console shows as the reason why the instance isn't active in the ELB. There does appear to be some kind of interaction between ELB and EC2 where ELB has some kind of awareness of an instance's EC2 state (e.g. "stopped") that goes beyond just the health checks. This isn't well-documented, but I would speculate that ELB, based on that awareness, decides that it isn't worth bothering with the health checks, and the console may provide something useful to at least confirm this.

It's possible that, given sufficient time, ELB might become aware that the instance is running again and start sending health checks, but it's also possible that instances have a hidden global meta-identifier separate from i-xxxxxx and that a stopped and restarted instance is, from the perspective of this identifier, a different instance.

...but the answer seems to be that stopping an instance and restarting it requires re-registration with the ELB.