After a few hours banging my head on the table, my colleague found the answer:
Why isn't factory_girl operating transactionally for me? - rows remain in database after tests
The factories turned out not to be transactional. The issue disappeared after the following line to spec_helper.rb:
config.use_transactional_fixtures = true