So I have managed to get the original installers working. My installer had a dependency for 'my' old .Net 3, which was why it wasn't working before. This is under the "launch condition editor" on the right-click menu of the installer, nb you also need the prerequisite on the properties to have .Net 4). After fixing the installer and using it to setup the website the system authenticates correctly. This leads me to believe that it was likely a problem with my configurations.
As to what property it was exactly I can't say, as comparing the web.configs has multiple differences. Why this IIS 7.5 works but it's web.config in IIS 8 doesn't is beyond me (same web.config).