On your Poll model, add a ManyToManyField to User, representing which users have voted in which polls. For each poll you want to display, check if this poll is in the current user's list of polls. If it is, do not allow them to vote.
And when the current user votes in a poll, add that poll to the current user's list of polls.