So I found out that the way to do this is to detect a meta key mask too, as depicted below.
getInputMap(WHEN_IN_FOCUSED_WINDOW).put(KeyStroke.getKeyStroke(KeyEvent.VK_META, InputEvent.META_DOWN_MASK), "meta-down");
getActionMap().put("meta-down", new MetaDownAction(this));