Vanity dashboard continues to update after experiment stopped

[chrislo]

trafficstars

I clicked the "make this option permanent" button next to an option in the vanity dashboard after I was satisfied that I had enough participants to make the decision. The experiment changed state in the dashboard (the finished timestamp is visible at the bottom), but checking it again today I see that the numbers continue to update (the selected option now has more participants). Is that expected behaviour? I expected the experiment results to "freeze" at the point where I stopped the experiment.

I am using the javascript registration and a redis backend, if that is relevant.

[phillbaker]

Hi @chrislo, thanks for reporting this, some more information would help:

• What version of vanity are you using?
• Did you confirm when you hit that "make permanent" button?

"Make permanent" should assign all current and future participants to the chosen alternative, so the numbers for the chosen alternative might increase, but the other alternatives should stop changing. (That chosen alternative should be shown/used by all users.)

On Wed, Feb 26, 2014 at 11:36 AM, Chris Lowis [email protected]:

I clicked the "make this option permanent" button next to an option in the vanity dashboard after I was satisfied that I had enough participants to make the decision. The experiment changed state in the dashboard (the finished timestamp is visible at the bottom), but checking it again today I see that the numbers continue to update (the selected option now has more participants). Is that expected behaviour? I expected the experiment results to "freeze" at the point where I stopped the experiment.

I am using the javascript registration and a redis backend, if that is relevant.

Reply to this email directly or view it on GitHubhttps://github.com/assaf/vanity/issues/195 .

[chrislo]

Hi @phillbaker, thanks for taking a look!

I'm using Vanity 1.9.0, and did hit the confirm button. I can confirm that the numbers for the alternatives do not increase, but the chosen one does. My use case was to run the experiment until I had N observations, stop the experiment, chose an alternative temporarily, and then report the results to my team. As all of the statistics on the dashboard are calculated using new data, there's no way to report what the "actual" results of the experiment were, if that makes sense?

It'd be great to have a section at the bottom of the experiment that said something like "This experiment finished on 27th February. At that time, option A (N observations) converted with X% and option B (M observations) with Y%. The result wasn't significant".

At the moment, the confidence reports are updating with the new data, and as one option is getting all the traffic, it now reports the other option as being the clear winner.