Requests going through EC2 load balancer hangs

StackOverflow https://stackoverflow.com/questions/23387728

Question

I have setup a simple proxy on an EC2 instance using Tinyproxy (default config listening/allowing all incoming connections). This works well. If I, for debugging, fill in IP address and the port to proxy settings in my browser, I can brows through the proxy without any issues. Everything works. However if I create an EC2 load balancer in front of the instance (making sure to forward the http port correctly) it just hangs when I browse through the load balancers IP. This seems like a puzzle to me. The instance is running, the load balancer reports "in service", and going around the load balancer works, but going through it just hangs. What am I missing out, and where should I look for the error?

UPDATE

I have now had a look at the logs of Tinyproxy: When trying to access google.com directly through the instances proxy, I see logs like this:

CONNECT   Apr 30 20:41:33 [1862]: Request (file descriptor 6): GET http://google.com/ HTTP/1.1
INFO      Apr 30 20:41:33 [1862]: No upstream proxy for google.com
CONNECT   Apr 30 20:41:33 [1862]: Established connection to host "google.com" using file descriptor 7.
INFO      Apr 30 20:41:33 [1862]: Closed connection between local client (fd:6) and remote client (fd:7)
CONNECT   Apr 30 20:41:33 [1901]: Connect (file descriptor 6): x1-6-84-1b-ADDJF-20-07-92.fsdfe [430.12327.65117.615]
CONNECT   Apr 30 20:41:33 [1901]: Request (file descriptor 6): GET http://www.google.ie/?gws_rd=cr&ei=_V9hU8DeFMTpPJjygIgC HTTP/1.1
INFO      Apr 30 20:41:33 [1901]: No upstream proxy for www.google.ie
CONNECT   Apr 30 20:41:33 [1901]: Established connection to host "www.google.ie" using file descriptor 7.

However if i try to access google through the load balancer, that then forwards to the instance, then I see logs like this:

CONNECT   Apr 30 20:42:54 [1860]: Request (file descriptor 6): GET / HTTP/1.1
CONNECT   Apr 30 20:42:54 [1869]: Connect (file descriptor 6): ip-432-2383245-53.eu-west-1.compute.internal [10.238.155.237]
CONNECT   Apr 30 20:42:54 [2037]: Connect (file descriptor 6): ip-432-2383245-53.eu-west-1.compute.internal [10.238.155.237]
INFO      Apr 30 20:42:54 [1860]: process_request: trans Host GET http://google.com:8888/ for 6
INFO      Apr 30 20:42:54 [1860]: No upstream proxy for google.com
CONNECT   Apr 30 20:43:12 [1861]: Connect (file descriptor 6): ip-432-2383245-53.eu-west-1.compute.internal [1230.23845.515.2537]
CONNECT   Apr 30 20:43:12 [2035]: Connect (file descriptor 6): ip-432-2383245-53.eu-west-1.compute.internal [143.238.12345.117]
ERROR     Apr 30 20:43:12 [2035]: read_request_line: Client (file descriptor: 6) closed socket before read.
ERROR     Apr 30 20:43:12 [1861]: read_request_line: Client (file descriptor: 6) closed socket before read.
ERROR     Apr 30 20:43:12 [2035]: Error reading readble client_fd 6
ERROR     Apr 30 20:43:12 [1861]: Error reading readble client_fd 6
WARNING   Apr 30 20:43:12 [2035]: Could not retrieve request entity
WARNING   Apr 30 20:43:12 [1861]: Could not retrieve request entity

From what I notice, then the ELB is trying to send the request through port 8888

Was it helpful?

Solution

You can get ELB access logs. These Access logs can help you determine the time taken for a request at different intervals. e.g:

  1. request_processing_time: Total time elapsed (in seconds) from the time the load balancer receives the request and sends the request to a registered instance.
  2. backend_processing_time: Total time elapsed (in seconds) from the time the load balancer sends the request to a registered instance and the instance begins sending the response headers.
  3. response_processing_time: Total time elapsed (in seconds) from the time the load balancer receives the response header from the registered instance and starts sending the response to the client. this processing time includes both queuing time at the load balancer and the connection acquisition time from the load balancer to the backend.

...and a lot more information. You need to configure the access logs first. Please follow below articles to get more understanding around using ELB access logs:

  1. Access Logs for Elastic Load Balancers
  2. Access Logs

These logs may/may not solve your problem but is certainly a good point to start with. Besides, you can always check with AWS Technical support for more in depth analysis.

OTHER TIPS

It sounds like you're trying to use ELB in HTTP mode as, essentially, something resembling a forward proxy, or at least a gateway to a forward proxy that you're running behind it. That's not ELB's intended application, so it isn't surprising that it wouldn't work in that configuration.

ELB in HTTP listener mode is intended to be used as a reverse proxy in front of a web endpoint/origin server.

Configuring your ELB listener to use TCP mode instead of HTTP mode should allow your intended configuration to work, but this is somewhat outside the optimum application of ELB.

http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-listener-config.html

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top