[Question] What is the Powered effect in the results?

If you could tell me. I can't fully understand it from the code

If you have set the power to, say 80%, then if the true difference between control and treatment was equal to the powered effect, then you would detect that difference (I.e. get a statistically significant result) 80% of the time.

This has a couple of implications:

• If the power is set to 50% and the measured effect equals the powered effect, then the lower confidence bound will be exactly zero, i.e. the powered effect is the smallest difference that will be statistically significant (but only when the power is set to 50%).
• The higher you set the power, the larger the powered effect will be since higher power basically means that you want to be more certain that the null is rejected when the true difference equals a specific alternative hypothesis.

Hope this help!

Thanks for the explanation. To be sure I understand the practical use, so if abs(difference)>=powered_effect then I can say my test has achieved or surpassed the power set as objective right?

Just curious, is there a documentation website for this package?

Thanks for the explanation. To be sure I understand the practical use, so if abs(difference)>=powered_effect then I can say my test has achieved or surpassed the power set as objective right?

Well, yes, but the usual way to use power is to use it when you, before launching the experiment, calculate the sample size. The power is one of the parameters that determine the required sample size, along with the significance level alpha, the minimum detectable effect size (MDE) and the variance of the metric in question.

The powered effect in the dataframe you get from the difference methods is the currently powered effect, based on the current sample size, which might be larger or smaller then the sample size required to power the MDE you set before launching the experiment.

Hope that clarifies it further!

Just curious, is there a documentation website for this package?

Not at the moment no. The only documentation we currently have is the doc strings and the example notebooks in the repo.

It would be good to have a more comprehensive documentation website.

Another question, is the "non_inferiority_margin" equivalent to "minimum detectable effect size (MDE)"? Is it in absolute or relative sense? I didn't see clear documentation on that.

It would be good to have a more comprehensive documentation website.

Yes, totally agree! I will see if we can find time to improve the documentation.

Another question, is the "non_inferiority_margin" equivalent to "minimum detectable effect size (MDE)"? Is it in absolute or relative sense? I didn't see clear documentation on that.

MDE are used to say what size improvement you want to be able to detect. NIMs are used to say what the largest deterioration of a metric you can accept, e.g. "I want to be sure (e.g. with 99% confidence) that latency doesn't increase more than 5%". Bot are given in relative terms, relative to the average. So, for example, if you have average latency of 200ms and then providing a NIM of 0.05 means that you want to make sure it doesn't go above 210ms.