Django A/B Split Testing Packages (None I've found are well-documented and up-to-date.)

Question

There are two main schools of thought for doing A/B (Split) Testing:

• Javascript-based solutions such as Optimizely, Google Analytics Content Experiments.
• Server-side solutions such as Django-AB, Splango, and django-lean. (Also, writing your own.)

My understanding is that Javascript-based solutions are spectacular for "which color button converts better," but not so great for switching out entire page layouts, and completely unworkable for trying out large functional changes such as the sequence of pages in a funnel.

That leads me towards a server-side solution. I'm not crazy about coding my own, and will do so only if there is no other option. I'm trying to add value by improving the core functionality of my site, not by creating a better split-testing framework.

The Django apps I've found for split testing are various mixtures of unmaintained, undocumented, documented incorrectly, and incompatible with Django 1.5. This surprises me, because the Django and Python communities seem to have a strong focus on good documentation. I'm also very surprised that none of the testing frameworks I've tried has been compatible with Django 1.5 -- is testing not as core a part of the philosophy in the Django/Python world as it is in Rails?

Here's what I've found:

• Splango https://github.com/shimon/Splango -- Not compatible with Django 1.5 (although most compatibility bugs I found were trivial to fix). Largely un-touched since October 2010, except for a fix August 2012 which claims to make sure templates get included in the install. Since templates don't get included in the install when Splango is installed via PyPI, either the fix didn't work or didn't get submitted to PyPI. Documentation is largely accurate, but doesn't completely cover how to set up tests and get reports. It tells you how to configure the template to gather the data, but there appears to be additional steps required in the admin interface which are completely undocumented, and I'm not sure I've done them properly.

• Django-lean. Original at https://bitbucket.org/akoha/django-lean has not been updated since July 2010. There is an apparently "blessed" fork at https://github.com/anandhenry2002/django-lean which has not been changed since May 2012, when it was copied over from the original. The original's documentation is incorrect in ways that make following the examples impossible. (Though you can probably muddle your way through, as I did.) The new version's documentation has formatting problems that make it difficult to read on github. (This appears to be because it's the unchanged documentation from the old project, and BitBucket syntax doesn't work on Github.) The django-lean Google Group has not had a message since July 2012.

• django-mini-lean https://github.com/DanAncona/django-mini-lean -- Updated as recently as February 2013, but undocumented.

• Leaner - https://bitbucket.org/brianjinwright/leaner -- Last updated July 2012, and no docs.

• Django-AB -- Last updated May 2009. Is not a package, and can't be installed via PIP or PyPI. After placing the checkout in my django app folder (and renaming the folder to ab) and following the installation instructions, I get an error loading the template loader that I have not tracked down further.

So far Splango appears to be the winner, as I've actually been able to get it more-or-less working (by manually installing the templates, and then editing them to fix Django 1.5 incompatibilities).

Can anyone point me to anything I've missed?

Answer 1

You have missed this app : https://github.com/mixcloud/django-experiments + https://github.com/disqus/gargoyle/

Answer 2

And then there's waffle: http://waffle.readthedocs.org/

It's simple, updated, maintained, but not very feature rich, it doesn't have any analytics/reporting stuff integrated. But then again, google analytics or mixpanel type of service is better for this.

Answer 3

I first looked at Django-AB and that is almost what I wanted, but I couldn't get it to work either. After looking at django-experiments and deciding I didn't want to mess around with redis yet, I decided to roll my own. I've tried to package it up nicely and make it easy to use for the beginner. It's super basic.

https://github.com/crobertsbmw/RobertsAB

Answer 4

You can swap out entirely different page layouts with Google Analytics Experiments (their default experiment setup will redirect users to a different URL for each variation you have), although in general its much easier to interpret why something is more successful if you test smaller things against each other.

You are right that testing different funnels and user flows against each other using Google Analytics would require a lot of manual setup; although theoretically you could do it by swapping out different links and tracking your users with UTM campaigns.

For smaller A/B tests within the same page, I ended up using Google Analytics Experiments and writing a custom Django CMS plugin for adding a few variant options to a template, which queries the Google Analytics API and displays the correct variant using Javascript.