Your problem is not related to JSoup, but to your rtfToHtml function.
Your function does not generates the <sub> and <sup> tags you expect. JSoup cannot do anything a this step, because the expected tags are not here, so you cannot parse them.
EDIT: (and Solution)
You should skip your rtfToHTML step when not necessary. If clipboard contains the data already in HTML format, so asking for it in rtf then converting it back to HTML implies losses of format information during conversions.
You can get clipboard directly in HTML format to avoid unnecessary conversions:
DataFlavor dfHTML = new DataFlavor("text/html; charset=Unicode");
boolean hasTransferableHTMLText = (contents != null) && contents.isDataFlavorSupported(dfHTML);
if (hasTransferableHTMLText)
{
InputStream is = (InputStream)contents.getTransferData(dfHTML);
String htmldata = org.apache.commons.io.IOUtils.toString(is, "Unicode");
Document doc = Jsoup.parse(htmldata);
System.out.println(doc.html());
//...
}
Tested with copy-to-clipboard from Chrome and FF. Both keeps the <sub> and <sup> tags you expect.
EDIT2:
IOUtils refers to org.apache.commons.io.IOUtils