I finally, after 5+ hours of googling and trial and error, have a working NPP Exec script that will compile and run a java program without leaving notepad++.
NPP_SAVE
cmd /K (javac "$(FULL_CURRENT_PATH)" && exit) || exit
cmd /K (cd /D "$(CURRENT_DIRECTORY)" && java $(NAME_PART) && exit) || exit
The only thing left would be finding a way to do the above, without having to call and send parameters to cmd, all in notepad++ and nppexec.
As noted in the comment below, if you're using a package, you will need to edit the second line accordingly. If your package name is the same as your file name, the below should work:
cmd /K (cd /D "$(CURRENT_DIRECTORY)" && java -cp .. $(NAME_PART).$(NAME_PART) && exit) || exit