Best practice for assigning A/B test variation based on IP address
-
22-09-2019 - |
Question
I am starting to write some code for A/B testing in a Grails web application. I want to ensure that requests from the same IP address always see the same variation. Rather than store a map of IP->variant, is it OK to simply turn the IP address into an integer by removing the dots, then use that as the seed for a random number generator? The following is taking place in a Grails Filter:
def ip = request.remoteAddr
def random = new Random(ip.replaceAll(/\./, '').toInteger())
def value = random.nextBoolean()
session.assignment = value
// value should always be the same for a given IP address
I know that identifying users by IP address is not reliable, and I will be using session variables/cookies as well, but this seems to be useful for the case where we have a new session, and no cookies set (or the user has cookies disabled).
Solution
You could simply take the 32-bit number and do ip mod number_of_test_scenarios. Or use a standard hashing function provided in ruby. But I feel I should point out a few problems with this approach:
- If your app is behind any proxies, the ip will be the same for all the users of that proxy.
- Some users will change IPs fairly frequently, more frequently than you think. Maybe (as Joel Spolsky says) "The internet is broken for those users", but I'd say it's a disservice to your customers if you make the internet MORE broken for them, especially in a subtle way, given that they are probably not in a position to do anything about it.
- For users who have a new session, you can just assign the cookie on the first request and keep the assignments in memory; unless a user's initial requests go to multiple servers at the same time this should resolve that problem (it's what I do on the app I maintain).
- For users with cookies disabled, I'd say "The Internet is broken", and I wouldn't go to much trouble to support that case; they'd get assigned to a default test bucket and all go there. If you plan to support many such users in a non-broken way you're creating work for yourself, but maybe that's ok. In this case you may want to consider using URL-rewriting and 302 redirects to send these users down one scenario or another. However in my opinion this isn't worth the time.
- If your users can log into the site make sure you record the scenario assignments in your database and reconcile the cookie/db discrepancies accordingly.