Finally figured this one out - turns out all you need to do is flip the TextMatrix
and it doesn't matter if you do it before or after you flip the coordinate system for the entire context. The reason why it didn't work previously was because I was applying it to the existing transform instead of assigning a new one i.e. this works
context.TextMatrix = CGAffineTransform.MakeScale(1.0f, -1.0f);
context.TranslateCTM(0, height);
context.ScaleCTM(1.0f, -1.0f);
But this doesn't
context.TextMatrix.Scale(1.0f, -1.0f);
context.TranslateCTM(0, height);
context.ScaleCTM(1.0f, -1.0f);
Also one other downside is the text is always rendered upwards now instead of downwards so you always need to translate the height of the text if looking to draw on (x,y) and not under.