Google Experiments outperforming or not?

Question

Today I met some strange thing in Google Analytics. One of my experiments ended with the probability of outperforming original of 94.7 % and outperforming original by 52 % and still with status "No winner found."

Has somebody met this situation? Why is it so?

Answer 1

This means that while you do have versions that are better than your control, they aren't winning by enough to reach the 95% confidence interval. In other words, you have a winner but it isn't statistically significant, or there's some chance that the outcome of this test is random.

Without seeing the numbers of impressions and conversions its difficult to be more specific. My guess is that either the conversion rate for this test is very low, the amount of improvement in the winning versions is very small, or both.

You have a couple of options for how to proceed.

1. Repeat the test using only the control and the strongest variant. It's possible that by repeating the test with only two versions your test will reach significance.
2. Implement the strongest variant and find something else to test. This is likely what I'd choose. Ideally you want to identify page elements that will have the greatest impact on conversion rates.

Either way though, I would end the current test. At this point there's nothing to be gained by letting it run longer.