growthbook icon indicating copy to clipboard operation
growthbook copied to clipboard

Option to base metric conversion window ends on the last exposure date, not the first

Open jdorn opened this issue 3 years ago • 0 comments

Currently, if a user sees an experiment multiple times (i.e. has multiple rows in the experiment assignment table), we only consider the very first exposure date when looking for metrics.

As an example, consider the following stream of events for a user:

  1. 1/1 Experiment assignment
  2. 1/5 Metric conversion
  3. 1/7 Experiment assignment
  4. 1/8 Metric conversion

If the conversion window is 2 days, these metric conversions will be ignored right now since neither of them happened within 2 days of the initial assignment.

There are a few possible ways to treat this data:

  1. Use only the first assignment event for the conversion window (current behavior)
  2. Use both the first and last assignment event for the conversion window (any conversions from 1/1 to 1/10 would be included)
  3. Start a new conversion window after each assignment event (any conversions from 1/1 to 1/3 OR from 1/7 to 1/9 would be included)

There should be a new per-experiment setting to choose which option should be used. Option 2 should be easy to add, but option 3 may be difficult.

jdorn avatar Jun 21 '22 19:06 jdorn

Use both the first and last assignment event for the conversion window (any conversions from 1/1 to 1/10 would be included)

How would this work with Exclude In-Progress Conversions given that the conversion window could theoretically always be extended by another exposure event? Therefore, it is never 'safe' to say a conversion is not 'in-progress'.

judahrand avatar Aug 24 '22 09:08 judahrand

@judahrand yeah, that's a good point. Basically how the "In-progress" setting works now is it only includes exposure events that happen before (now - conversionWIndow). So if your conversion window is 3 days and you look at results on Aug 24th, it will only consider exposure events before Aug 21st.

I'll have to think through the implications of this PR a little more, but I don't think it will have too much of an effect on this behavior.

jdorn avatar Aug 24 '22 18:08 jdorn