Q: Variant consistently strong in the first week, but drops of to no change in the 2nd. Normal?

[christian314159]

Hi there,

we've been using the split gem for a while, but we've seen this behavior a couple of times and wonder if it's normal:

We run an A/B test where the new variant is very strong (~4-6% better than control) in the first week (with 100k+ participants), even shows up as 90%+ confidence in the dashboard, but then in the 2nd week (we usually try to run experiment for at least 2 weeks) it just drops off to no real change against the control.

Is this normal? Could we have done something wrong with the setup? Thanks!

[feliperaul]

Curious to see this issue, because we've noticed this behavior as well!