Could Vote, in this case be considered an "aggregate" and be deserving of its own repository and aggregate status?
I think this might be the right answer. An aggregate should be a transactional consistency boundary. Is there a consistency requirement between votes on a match? The presents of a Vote collection on a Match aggregate would suggest that there is. However, it seems like one vote has nothing to do with the next.
Instead, I would store each vote individually. This way you can use the aggregate functionality of MongoDB to get the count, though I'm not sure whether it is still slow. If it is, then you can aggregate using the Map/Reduce functionality.
More generally, this may not be a best fit for DDD. If the domain doesn't consist of complex behavior there is hardly a reason to try to adapt the DDD tactical patterns (entity, agreggate) to this domain.