Are you having large spikes of traffic? If so, please know that the ELB scales pretty slow (50% every 5 minutes). You can refer to this article for more info: http://aws.amazon.com/articles/1636185810492479.
If you have a relatively constant traffic usage, you have probably saturated the EC2 instance that you are currently using. In this case you should implement an auto-scaling policy, so that you have enough compute power for your current usage.
Basically, unless you have more traffic that the ELB can temporarily handle (this issue solves itself automatically as the ELB scales), this should not be an ELB issue. I've seen latencies as low as 100ms through elb, at over 100k RPM.