KeyEvents are only generated for the component that has focus. A JPanel is not focusable by default.
Don't use a KeyListener. Instead you should be using Key Bindings which are more flexible.
See Motion Using the Keyboard for more information and examples.