First, this statement could not compile, if Outer
is a class type:
if (!(Outer instanceof Outer.Nested)) // Outer is not an expression: Expression expected
Without no precised context in the question, I imagine you would deal with a scenario like this:
public class Outer {
private class Inner {
}
public static void main(String[] args) {
Test t = new Test();
Inner i = t.new Inner();
System.out.println(i instanceof Test); //inconvertible types => normal
System.out.println(t instanceof Inner); // inconvertible types => normal
}
}
All happens normally thus.
If your scenario is similar: sounds like an issue with your compiler process.
If your scenario isn't similar: please update your question with more information.