I have discovered a work-around.
1) Go to the location in which Visual Studio caches the app_offline.htm file C:\Users[user]\AppData\Roaming\Microsoft\VisualStudio\11.0\ (replace [user] with your username)
2) Delete the app_offline.htm file
(NOTE: this alone will not solve it, as VS regenerates the file again when it's not found)
3) Create a directory called app_offline.htm (a DIRECTORY!)
This essentially causes an error with the process that attempts to use the file because it's not a file and the name clash means VS cannot try to recreate the file. Luckily the error has no unwanted side-effects.
Now continue with using VS and Team Foundation Server and enjoy not constantly having the appdomain reloaded!