Modality works only for windows, so you can't use it to block tabs unless you overhaul the mechanism (and it's a non-trivial one).
One option is to call setEnabled(false)
on the tab and component in its view - the content pane of the frame in your case. The implementations depends on if the dialog pops when you do something in a tab, if it pops regardless of what tab you are in or some other way.
You can call setEnabled
in an override of setVisible
of the dialog - which should give you the desired results (this is not where modality is called, so it might behave slightly differently in certain circumstances).
Another option is to add some mouse listener to manually block clicks on the tab and \ or its component-in-view. This is more messy but resembles more how modality acts.
As to the modality of the dialog, it depends if you want it to block its parent frame - the one whose content pane is displayed in the tabs. Note that there is no relation between the tab and the frame the way you present it.