So, based on the helpful comments I managed to figure it out.
The src.zip given by the jdk is incomplete, if it doesn't contain the class you need to modify, try going with openjdk.
While the best practice would be to use -Xbootclasspath with a directory with your modified files, it simply does not work (or is too cryptic to figure out). There is however, jce.jar which contains javax.crypto somewhere in a hidden jre/lib directory. You should unzip the jar, replace the compiled class files and rezip everything. This finally works :D