I've finally found a way which works for me. The following method removes the mailmerge settings:
private void removeMailmergeRelationship(WordprocessingMLPackage wordMLPackage) {
DocumentSettingsPart dsp = wordMLPackage.getMainDocumentPart().getDocumentSettingsPart();
CTSettings settings = dsp.getJaxbElement();
settings.setMailMerge(null);
}