It looks like there is an open bug about this in the project.
https://github.com/perfectline/validates_existence/issues/15
You may have to write a simple custom validator for this case until this gets fixed. (or dig in and see if you can fix the issue yourself)
UPDATE: I just cloned the project and wrote a test to see what the issue was. It seems that when you add the allow_nil, the existence validator doesn't get called at all. I'm not sure why that is, but in the meantime, you can work around the bug in an easy way, by using a proc. instead of
validates :contributor, :existence => { :allow_nil => true, :both => false }
this would get the job done
validates_existence_of :contributor, :unless => Proc.new { |obj| obj.contributor_id.blank? }
I was able to prove that in my test case. (I went with the 'validates_existence_of' method, instead of 'validates', because I thought it was cleaner in this case)