If you want/need to stick with your current schema, one way of preventing duplicate votes by IP Address is that you could use something like memcached(very fast) and for each vote store the question id and the ip address. If your using Doctrine as your ORM then you could use a preUpdateEvent to check for the existence of the ip address.
You could setup a custom Validation Constraint on your entity to check for the ip address as well which has the benefit of presenting friendly messages to your forms.
Another possibility is to setup an Event Listener in your controller or model manager to check for the ip address.
Hope that gets you going in some decent direction.