Thanks to @assylias for pointing me to rlwrap
.
I installed rlwrap via homebrew:
brew install rlwrap
Then I defined an alias in my .zshrc (would be the same for bash in .bashrc)
alias jjs='rlwrap /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/jjs'
Now I can use jjs
on the commandline and the <- and -> keys are working as expected.