I assume you are using GUIDE.
This is related to resize behaviour. If you follow Tools-> GUI Options you can see the combobox for resize behaviour. Yours is probably set to "Proportional".
If you change it to "Non-resizable", you will be able to resize your panel without affecting children sizes. You can change it back to "Proportional" after you are done.