Well, to close my own question after so much.
In the end the problem was ADFS naming of Relying Parties, once we switched the name to an URL (which took some convincing) it started working.
ADFS should be string in the name format for the RP identifier.