A/B testing split visitors equally

Question

I am writing and testing an a/b test library. I am splitting visitors between variations using a Math.Random(...) call.

However, this isn't working well. I've had over 20% difference in displays on variationA vs variationB.

Does anyone know a better option to do this, other than comparing the displays and, if difference is larger than X%, to decide on showing the other variation without calling Math.Random?

Answer 1

If you want to keep things more random than the A/B/A/B/A/B answer, you could skewer the odds of your randomly generated A or B towards the group with the least members.

For example, if Random returns a number between 0 and 1, you would have something along the lines of:

variation = if(Random() > 0.5) { variationA } else { variationB }

If you change this to (nrOfA/B = times A/B was chosen):

variation = if(Random() > (nrOfB / (nrOfA + nrOfB))) { variationA } else { variationB }

Then the likelyhood of A or B being chosen will depend on how big they are compared to the other. The bigger B is compared to A, the bigger the chance A will be chosen and vica versa.

Answer 2

If you would like to be 100% sure to have an equal number of views between A and B, you should store in cache/db on which variation was assigned the last variation. Then on each request, assign the other variation. This way you will have A/B/A/B/A/B etc.