Beta testing new website features with live data and real customers

Question

The main goal of this question is to determine the pitfalls of deploying a slightly modified version of a website alongside a live website.

This secondary website would be pulling from the same database as the live but would have modified features for beta testers.

The end goal is to allow certain customers test our new features with their data.

So:

1. They don't have to do things twice by going to a copied version of the site.
2. They are using familiar data sets

Another possibility would be setting a flag per user account to allow them to see certain features but this would require a lot of extra work. Also, once it is ready for release, we would have to remove all the extra checks.

I am having a hard time seeing the disadvantages of this, but I know there has to be some glaring at me. Thank you for any assistance.

Git version controlled, Capistrano Deployment workflow, Cakephp framework, MySql We currently have local and testing servers that are separate from our production servers.

EDIT 12-20-2012 10:30am EST

Based on some comments and one answer I have an update based on feedback.

1. Meticulous internal testing should be done before 'beta'/user feedback testing. (which we already do)
2. If we take these precautions and the code base seems solid, the risk in deploying alongside the production server could be manageable. We are working within a framework here so the likelihood of mass deletion and bad sql is relatively low.

All that being said, I would rather not take this approach because it still has inherent risk. Does anyone do beta testing with live server data another way?

Answer 1

It depends...

If this is a beta to get customer feedback, on a product that has been fully tested and is known to be stable, the risks are relatively manageable (though see points below). This is the way Google defines "beta".

If "beta" means code complete, and sorta-kinda tested, but who knows what bugs are in there, you risk corrupting your live database. No matter how clever your backup strategy, if something goes wrong, the best case scenario is that the beta users face data loss or corruption; the worst case is that all your users lose data (I've seen broken "where" clauses in delete or update statements do all kinds of entertaining damage).

Another issue to consider is whether the database is backward and forward compatible between versions - can you migrate your beta users back to the mainstream version if they don't like the upgrade, or if something goes wrong? This is a far bigger deal if "beta" means "untested", of course.

In general, it's a lot easier to deal with one-way compatibility - allowing users to upgrade, but not downgrade - another strong argument for "beta" to mean "user feedback"...