I found the solution the actual exception is nested deep in the compositionException.Errors hierarchy. One would think the original exception would be under the original composition exception but actually this is how to access it:
It happens to be the innerexception inside one of the errors of another composition exception which is itself one of the errors in the original composition exception
CompositionException compositionException = (CompositionException)activationException.InnerException;
CompositionException innerCompositionException = compositionException.Errors[0].Exception;
InvalidCastException castException = (InvalidCastException)innerCompositionException.Errors[0].Exception.InnerException
I won't delete this in case someone else runs into the same thing.