In my experience, this can happen if a disproportionate amount of traffic is coming from a single network or ip address.
ELB uses different layers of balancing. DNS load balancing will send it to a set of IP addresses in one of the two zones, and the software load balancer will distribute traffic between instances in the zone.
If you have a lot of traffic coming from the same network, its likely that a lot of users are getting the same DNS resolution on your load balancer and ending up in the same zone.