There could be a number of reasons for this. Its without doing more digging, its hard to know which one you are experiencing.
Sticky sessions can result in instances traffic becoming unbalanced. Although this depends heavily on usage patterns and your application.
Cached DNS resolution. Part of how the ELB works is to direct traffic round robin on a DNS level. If a large number of users are all using the same DNS system provided by an ISP, they might all get sent to the same zone. Couple this with sticky sessions and you will end up with a bunch of traffic that will never switch. Using Route 53 with ALIAS records may reduce this somewhat.
If you can't get the ELB to balance your traffic better, you can set up something similar with vanish cache or other software load balancer. Not as convenient, but you will ultimately have more control.