Looks like it is definitely possible. Check the MSDN link and look at the topic "Multiple Build Server system". In the picture it show a single collection which is connected to multiple build controllers. Definitely possible.
http://msdn.microsoft.com/en-us/library/dd793166.aspx
The video below also give you a step by step process on how to do it.