I've been looking at this and I have a workaround, if you keep a reference to all the tags you want to keep, remove the tags and add the ones you want back in, the file size shrinks. I realize this is pretty horrible, but it works. So if you just want to keep the title:
string title;
using (var taglibFile = TagLib.File.Create(file))
{
title = taglibFile.Tag.Title;
taglibFile.RemoveTags(TagTypes.AllTags);
taglibFile.Save();
}
using (var tagLibFile = TagLib.File.Create(file))
{
tagLibFile.Tag.Title = title;
tagLibFile.Save();
}