Simpler is better. You should refactor if you don't need the extra model.
The simplest rule of thumb is that you should set up a has_many :through relationship if you need to work with the relationship model as an independent entity. If you don't need to do anything with the relationship model, it may be simpler to set up a has_and_belongs_to_many relationship (though you'll need to remember to create the joining table in the database).
From the guides.
I don't think you have to worry about performance at this level, just take the solution that fits your needs.