It turns out that I had to change the tfs build settings for msbuild to build each of the projects one at a time.
Here are the steps to take:
Goto builds -> Edit Build Definition -> Process Tab -> Edit "MSBuild Multi-Proc" to False
Not the best solution as it slows the total build process down because the projects cannot build in parallel but it worked