Amazon Load Balancer excessively high latency

Question

I'm having an issue with an AWS load balancer - loading pages through it seems to give high latency (~5s)

There are two EC2 instances living behind the load balancer, let's call them p1 and p2.

I'm running Magento on these instances, they're both connected to the same database.

When viewing a category page on p1 or p2 directly, the initial load time is < 500ms, but when I visit the load balancer (which then points to p1 or p2) the browser spends ~5 seconds waiting for a response from the server.

This is a typical request to p1 or p2 directly:

This is a typical request from the load balancer:

I initially suspected it may be an issue with Magento trying to re-cache for requests coming from the load balancer but I then set p1 and p2 to have their caches synchronised so cache is unlikely the cause.

The stack on p1 and p2 are fairly regular Apache2 + PHP-FPM + PHP setups that are lightning fast on their own.

Answer 1

AWS has recently released a new feature of ELB just for such troubleshooting scenarios. Now you can get ELB access logs. These Acces logs can help yopu determine the time taken for a request at different intervals. e.g:

1. request_processing_time: Total time elapsed (in seconds) from the time the load balancer receives the request and sends the request to a registered instance.
2. backend_processing_time: Total time elapsed (in seconds) from the time the load balancer sends the request to a registered instance and the instance begins sending the response headers.
3. response_processing_time: Total time elapsed (in seconds) from the time the load balancer receives the response header from the registered instance and starts sending the response to the client. this processing time includes both queuing time at the load balancer and the connection acquisition time from the load balancer to the backend.

...and a lot more information. You need to configure the access logs first. Please follow below articles to get more understanding around using ELB access logs:

1. Access Logs for Elastic Load Balancers
2. Access Logs

These logs may/may not solve your problem but is certainly a good point to start with. Besides, you can always check with AWS Technical support for more in depth analysis.