Jacob,
It sounds like you're overloading your system. With any significant load run, regardless of the toolset, you need to get the various load components distributed out to run on different systems.
If you haven't already, you should have a look at the Working With Load Tests documentation on MSDN. See also the Considerations for Large Load Tests (especially the part about overloading Agents) as well as the Run a Load Test Using Agents pages.