This question hints at a lot of problems and possible solutions. The simplest answer is probably to just add more build servers to run all the builds in parallel.
Otherwise you need to consider turning off rolling builds and writing your own build scheduler. That or other strategies such as building the DLL once and checking it in as source to the other builds or running the build on a branch that is outside of the other 5 builds and merging the source into those builds when they want to pick up the change.
If you are otherwise happy with how things are now and don't want to do a lot of work to solve the problem.... then just scale out your build farm with additional agents.