So I think I determined what this issue was. I had a secure channel being used first in the list of channels, and since it was different domain (http instead of the https that the channel was on) it would do the crossdomain.xml lookup.
I reordered the channels to have the regular http channel first, and my application no longer does any crossdomain lookups except the forceful one (which I've since removed).
Since the first channel was over https, Flex was waiting on that lookup before falling back to the insecure channel.
I also think this should continue to work if actually using https, since I believe that a plaintext request to a secure connector will be rejected instantly instead of timing out.