You don't have to declare a listener class for each button, you can just use the same one and add if statement to determine from which button the action come from.
if (e.getSource() == blueButton) {// e is the ActionEvent
blueButton.getParent().setBackground(Color.BLUE);
} else if(e.getSource() == redButton) {
redButton.getParent().setBackground(Color.RED);
}