How to determine if a company decision was successful or not?

Question

I'm trying to figure out if a decision taken in a company (offering discounts for specific products) is successful or not. I have done some research and saw that A/B testing might be a way to do this but A/B tests can only be carried out having 2 groups (control and experiment), and in this case all I have is past and "after the decision" data.

Can the statistics behind AB testing be used in this case?

The data I have is in the for of sales per day per article only for one company before and after the decision (only one decision).

Example:

sales for articles X and Y from the 2016-01-01 to 2016-02-01. The decision was to lower discounts on articles A and B on 2016-01-15, so I want to know if sales decreased or not after this decision.

Answer 1

You could use time series approach to model the "what if not" scenario and compare it with your values after the new program introduction. Check the causal_impact packet from Google and you might also find helpful this tutorial for probabilistic programming. http://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Prologue/Prologue.ipynb

Answer 2

You are trying to determine if some event/treatment (i.e. going on sale) has a significant effect on sales of some product_x. An A/B test is appropriate to answer this question if you have both test and control groups.

Given: (T)est and (C)ontrol groups, where O1 = observation 1, a baseline measurement of normalized sales (i.e. average sales of product_x over some pre-defined time period) and O2 = after treatment measure of normalized sales. Between the first and second observation/measurement, T are given experimental treatment X.

Typical experimental design -

T: O1 X O2

C: O1 . O2

From the OP's description it sounds as if you don't have an adequate control group - although not ideal, it is possible to create control-like groups after the fact (sometimes you can use packages to help create matching control groups after the fact e.g., {optmatch}). In your case, is it possible to get sales data from a similar store to the one you are studying, but one which did not have the sale on product_x? If so, this may be adequate as a control group.

From your description, however, it sounds like you have a pre-test post-design:

Given: (T)est group with treatment X and normalized sales before and after the event

-----time----->

T: O1 X O2

With this design, the primary threat to validity is lack of an adequate control group to ensure that any observed effect is actually due to the experimental treatment (i.e. the Sale). For example, if you observe an increase in over all sales of product_x from the pre to the post-sale period, this finding is consistent with your original hypothesis (A Sale will increase overall sales of product_x), but it is also consistent with some sort of temporal effect that happens to correlate with sales (e.g. seasonality effects, where product sales increase during certain time periods like the holiday season).