BorderLayout not position component in correct position

Question

I'm adding and removing a JButton on MouseEnter and MouseExit respectively. This is working ok but when the button is added to the panel it appears in the top-right corner instead of the BorderLayout.SOUTH position specified.

The Frame only has a JPanel and the only line I have added is

 jPanel1.addMouseListener(new myMouseListener(jPanel1));

The Mouse Listener

package example;

import java.awt.BorderLayout;
import java.awt.Color;
import java.awt.event.MouseAdapter;
import java.awt.event.MouseEvent;
import javax.swing.JButton;
import javax.swing.JPanel;
public class myMouseListener extends MouseAdapter{
    JButton btn;
    JPanel panel;

    public myMouseListener(JPanel panel)
    {
        this.panel = panel;
        this.btn = new JButton("Test");
        this.btn.setSize(40, 40);
    }

    public void mouseEntered(MouseEvent e) {
        panel.setBackground(Color.red);
        panel.add(btn, BorderLayout.SOUTH);
    }

    public void mouseExited(MouseEvent e) {
       panel.setBackground(Color.blue);
       panel.remove(btn);
    }
}

You can download a sscce here
http://www.filehosting.org/file/details/302851/Example.zip

Can anyone shed some light on the issue?