Best Practice for Assigning A / B Test Variation Based on IP Address

Question

I am starting to write code for testing A / B in a Grails web application. I want to make sure that requests from the same IP address always see the same options. Instead of storing the IP-> option map, is it enough to simply translate the IP address into an integer by deleting the dots, and then use this as a seed for a random number generator? The following happens in the Grails filter:

def ip = request.remoteAddr def random = new Random(ip.replaceAll(/\./, '').toInteger()) def value = random.nextBoolean() session.assignment = value // value should always be the same for a given IP address 

I know that identifying users by IP address is not reliable, and I will also use session variables / cookies, but this seems useful for the case when we have a new session and the cookie is not set (or the user has cookies disabled.)

Answer 1

You can just take a 32-bit number and make ip mod number_of_test_scenarios. Or use the standard hash function provided in ruby. But I believe that I should point out a few problems with this approach:

• If your application is behind any proxies, ip will be the same for all users of this proxy.
• Some users will change IP addresses quite often, more often than you think. Maybe (as Joel Spolsky says) the Internet is broken for these users, but I would say that it’s harming your customers if you make the Internet MORE broken for them, especially in a subtle way, given that they are probably not in able to do something.
• For users who have a new session, you can simply assign a cookie on first request and save the destination in memory; if user initial requests do not go to multiple servers at the same time, this should solve this problem (this is what I am doing in the application that I support).
• For users with cookies disabled, I would say: “The Internet is broken,” and I would not worry about supporting this business; they will be tied to the default test bucket and everyone will go there. If you plan to support many of these users in a non-broken way, you are creating a job for yourself, but perhaps this is normal. In this case, you might want to use URL rewriting and 302 redirects to send these users in the same scenario. However, in my opinion, this is not worth the time.
• If your users can log in, make sure that you record the script assignments in your database and reconcile cookie / db discrepancies accordingly.