You can seal the package. That prevents people from adding classes to the classpath (though you still have to protect your jars).
http://docs.oracle.com/javase/tutorial/deployment/jar/sealman.html
Having sealed your jar, you can still compile a class which claims to be in the same package, but when you run it, it'll fail to run:
Exception in thread "main" java.lang.SecurityException: sealing violation: can't seal package org.splore.so.access: already loaded
at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:395)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:417)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.splore.so.access.AccessTestFromOtherPackage.main(AccessTestFromOtherPackage.java:5)