Your way of diagnosis is correct and yeah Web API's content negotiation process will try to find the best formatter based on bunch of logic(ex: Accept header if present, Content-Type header if Accept-Header not present, asks each formatter if it can serialize a type etc.).
You can disable this default behavior (i.e finding the first formatter in the list of formatters which can write/serialize a type) by doing the following. This will result in a 406 Not Acceptable
response being generated:
DefaultContentNegotiator negotiator = new DefaultContentNegotiator(excludeMatchOnTypeOnly: true);
config.Services.Replace(typeof(IContentNegotiator), negotiator);