I figured there was at least two ways of solving this:
Break the solution up per-project Nuget packages and build them in the proper environments. This is what I would have done in Maven land. While theoretically possible using MSBuild and Nuget, it's beyond manageable for me at this point.
Manually share the project files master->slave and execute relevant build commands on the slaves. This is what I ended up doing. Since my master runs Windows, I mounted the master workspace over smbfs on the slaves.