This behavior is implicitly documented in the POD for use.
Because "use" takes effect at compile time, it doesn't respect the ordinary flow control of the code being compiled. In particular, putting a "use" inside the false branch of a conditional doesn't prevent it from being processed.
The use statment is part of the parse tree of your code. It is executed during the compile phase of perl. The Perl compile won't add the use statement to the parse tree. This means that it won't be executed during the run time of the program.
If you are really curious how the code is parsed, then you can inspect the parse tree with B::Deparse.