You are only affecting the visibility of the injected-class-name. The access protection of the base subobject or its members should not be affected. If Clang or GCC allows it to affect a cast validity or access within the base, that's their bug.
[class.member.lookup] 10.2/3 says
In the declaration set, using-declarations are replaced by the members they designate, and type declarations (including injected-class-names) are replaced by the types they designate.
The base class subobject does not have a name in member lookup; the injected-class-name does.