I have solved this and I'm not sure if this an internal nant contrib issue or I just didn't do it correctly.
To solve this I had to create an exec task that pointed to the msbuild installed at C:\Windows\Microsoft.NET\Framework\v4.0.30319.
When i would run the msbuild task it used this path C:\Windows\Microsoft.NET\Framework64\v4.0.30319
So internally I think nant is defaulting to the 64 bit path on 64 bit machines.