The method names are a big hint, and the definitive answer is in the javadocs:
For Container.remove(Component)
:
"Removes the specified component from this container."
"Removes all the components from this container."
As to what is causing the difference in behaviour, my guess is that there is some other component in the panel that is getting removed in the first case, but not in the second case.
Try calling and logging / printing getComponentCount()
after the "remove" in both cases.