Classification model accuracy with ensemble methods

Question

I came through this statement in a Machine Learning text book based on law of large numbers:

Suppose you build an ensemble containing 1,000 classifiers that are individually correct only 51% of the time (barely better than random guessing). If you predict the majority voted class, you can hope for up to 75% accuracy!

I understand the analogy if we consider average over 1000 predictions but how majority votes lead to 75% accuracy from 51% (individual)?

Answer 1

This comes from the Binomial distribution, where you have $n=1000$ independent trials (models), $p=0.51$ of each model being right and since you care about the majority vote you want to have at least $k=500$ successful trials. That leads to:

$$\text{Pr}(k\geq500 \text{ models are right}) = \sum^{1000}_{k=500}\binom{1000}{k}0.51^{k}(1-0.51)^{1000-k}=0.74675\approx0.75$$

Here is how I calculated it:

    import numpy as np
    from scipy.stats import binom
    np.sum([binom.pmf(k,1000,0.51) for k in range(500,1001)])