A/B test on gathered data in database

Question

Assume I have a website with YES and NO buttons. Now, I log number of YES clicks and number of NO clicks for two days.

After having this data, I want to do A/B-test analysis on these data. So, I split data into 50-50 split. Use one as Control data and one for Test data. For Test data I count number of Yes and No clicks in 10 hours (instead of two days).

Do you think it is a good way to do A/B-test? As far as I know, I need two versions of website with slightly different feature while in my case website is identical for all users and I only split original data with some assumption. If it is not right, please refer me to an article or website.

Answer 1

You need the two groups of users to be identical (selected randomly) and so it is much preferred to run the groups in parallel.

For more info on how to interpret the results I highly recommend this blog: http://visualwebsiteoptimizer.com/split-testing-blog/, especially "What you really need to know about mathematics of A/B split testing" and "How large should your A/B test sample size be?" are must reads.

There is also a nice significance calculator available here.