Sample size for google content experiment

Question

Can anybody give me any idea about what kind of traffic / sample size I need to get a statistically significant result when doing a google content experiement for 2 variations?

Answer 1

Google uses Multi Armed Bandit testing. Here is a good article on this Googles answer

The best way in practice is to watch the percentage in the Google analytics experiments tab and see how quickly it moves toward 95%.

You can't get an exact answer because it changes as you take measurements and based on the difference you are trying to measure. So if one variation performs 300% better than the other it will take a lot smaller sample size than if one variation only performs 10% better than the other.

To see how the math for straight up statistical significance works here is a good explanation. Statistical significance tutorial

Here is a spot where it has a calculator Calculator

As far as the math for the Multi Armed Bandit this quote by Peter Whittle sums it up

[The bandit problem] was formulated during the [second world] war, and efforts to solve it so sapped the energies and minds of Allied analysts that the suggestion was made that the problem be dropped over Germany, as the ultimate instrument of intellectual sabotage.