enableModifiedFlag allows you to refine the functionality.
Custom Controls also have an enableModifiedFlag, which should allow you to give a message only if something in that custom control has changed.
Individual input controls have a disableModifiedFlag that can be set to true, to ensure that particular control is ignored when identifying if the page has been modified or not.
A Button of type "Cancel" will ignore enableModified and just move on.
You can also programmatically set or clear the modified flag in CSJS (XSP._setDirty(false,""). There is also a view.setEnableModifiedFlag(boolean) method that can be used to change the enableModifiedFlag property on an XPage from SSJS.
Panels don't have an enableModifiedFlag or disableModifiedFlag property, but with the options I mentioned it should give you the control you need.