require-abtest

require-abtest copied to clipboard

How do we expire A/B test cookies so that we can create another test?

Say we're testing /views/page.js via RequireJS. We run our tests like so:

define(['test!/views/page.js'], function(page) {
    // ...
});

If we stop running that test and want to run another test on page.js, the user will still have the cookie for the first test's cohort. We need a way to manage and expire tests.

A potential solution:

1. Track the date (yyyy-mm-dd) of when a user is assigned a variation in the cookie.
2. Add a "start date" to test definitions

If the start date is after the user's current variation assignment we can assume that we're running a new test and need to re-assign a variation.

After thinking about the solution above, it's a great first step but doesn't solve expiring A/B tests. This question should really focus on:

• Should we set A/B tests to run for a certain period of time (why?)?
• How do we expire A/B tests after statistically significant results?

Regarding question two, we're doing tracking entirely in JS. The example tracking package implements Google Analytics and GA Experiments. Expiring A/B tests after significant results just might not work.