CampaignPlanner_v3
CampaignPlanner_v3 copied to clipboard
Shiny demo of A/B test planning and evaluation (improved UI for A/B testing method taught in free video course)
CampaignPlanner_v3
Shiny demo of A/B test planning and evaluation
Improved user interface for A/B testing theory taught in this free video course Bayesian A/B campaign testing (and design).
Also available as a public running app.
Win-Vector LLC's free video lecture on Bayesian A/B testing
Source code: https://github.com/WinVector/CampaignPlanner/ App hosted online at: https://win-vector.shinyapps.io/CampaignPlanner/
Newer (better) online app: https://win-vector.shinyapps.io/CampaignPlanner_v3/ Newer (better) source code: https://github.com/WinVector/CampaignPlanner_v3/
The course emphasizes how to design A/B tests using prior "guestimates" of effect sizes (often you have these from prior campaigns, or somebody claims an effect size and it is merely your job to confirm it) using Bayesian tools.
The solution is coded in R and Nina Zumel has contributed an updated Shiny user interface demonstrating the technique.
This sort of fills out our survey of ways to think about A/B testing:
- Classic frequentist theory: emphasizes correct decision over expected returns/value. http://www.win-vector.com/blog/2014/05/a-clear-picture-of-power-and-significance-in-ab-tests/
- Bandit Formulations: great utility theory based solution to the problem. http://www.win-vector.com/blog/2014/04/bandit-formulations-for-ab-tests-some-intuition/
- Dynamic programming methods: more involved tracking of utility. http://www.win-vector.com/blog/2015/07/dynamic-prog-ab-test-design/
- Sequential analysis: operationalizing many of the above ideas. http://www.win-vector.com/blog/2015/12/sequential-analysis/
- Bayesian methods: correct distribution inference for individual test runs. (this course/playlist).
See here for another survey of our on the topic: http://www.win-vector.com/blog/2015/06/designing-ab-tests/