Limited selection in a JTextField/JTextComponent?

Question

Consider a JFormattedTextField (or any JTextComponent, really) wherein there is a prefix and a suffix displayed around what is the actual "text" of the field.

For instance, the double 3.5 would be the String (via formatting) "3.50" around which would be the prefix "$ " and the suffix "", for a display text of "$ 3.50".

Clearly, this is simple to do. However, the user is still allowed to select text within the prefix/suffix, so they could conceivably delete part or all of the prefix/suffix. I would prefer the user be restricted such that the prefix/suffix cannot be selected at all (while still part of the text field, so no JLabels). I can almost accomplish this with a CaretListener (or by overriding setCaretPosition/moveCaretPosition), which prevents a C-a from selecting the entire field, and it prevents using the arrow keys to move into the prefix/suffix. However, mouse dragging and shift-arrow keys still allows the selection to move into these restricted areas.

Any ideas?