Ok, I finally figured this out. After many many hours of head banging.
It seems that the ASP.NET MVC framework returns JSON as default it if cannot find a content-type or Accept header.
As SharePoint does not allow you to customise the headers for data sources, it was receiving JSON back as a default.
I Couldn't find a way to override this, so i had to remove the JSON formatter from my ASP.NET API as a temporary solution.
GlobalConfiguration.Configuration.Formatters.Remove(GlobalConfiguration.Configuration.Formatters.JsonFormatter);
Now, it returns XML regardless of the content disposition.
Edit:
I switched the formatters around, so if the Accept/Content-type is unknown, it defaults to XML.
var switch_it = GlobalConfiguration.Configuration.Formatters.JsonFormatter; GlobalConfiguration.Configuration.Formatters[0] = GlobalConfiguration.Configuration.Formatters[1]; GlobalConfiguration.Configuration.Formatters[1] = switch_it;