I do not have the rep to comment, but was going to ask you to post your factories as well.
Some other ideas: Take a look at how you can handle assocations with FactoryGirl. I'm assuming you are not on Rails 4 because of:
attr_accessible :queue_privacy_id, :user_id