Yep, found the answer myself. Envers is doing inserts for orphan deletes:
ISession session = sessionFactory.OpenSession();
ITransaction tx = session.BeginTransaction();
MyItem item = session.Get<MyItem>(44); //Item enthält ein Tag.
item.Tags.Clear();
tx.Commit();
session.Close();
But only if the relation end is not declared inverse.
HasManyToMany(x => x.Items)
.Table("MyItem_MyTag")
.ParentKeyColumn("MyTagID")
.ChildKeyColumn("MyItemID")
.Cascade.All()
.Inverse().LazyLoad();
If you want an entry non the less, then add:
nhibernate.envers.revision_on_collection_change