Just a thought... With MTUs at that size you start heading towards the memory page limit, though admittedly at 5000-5050 you should have eclipsed it already for a 4K page size (default for Xen 3.0). Still, I'd just wonder if your memory is getting fragmented. Try upping your memory page size so that you are sure that what you want to fill in a frame will fit in one page of memory and see if that helps.
... Actually, the answer to your question might be here: http://comments.gmane.org/gmane.network.iperf.user/485