cluster-experiments
cluster-experiments copied to clipboard
feat: Add Variance Reduction to Delta Method and couple it with CUPAC
CUPED is used with the Delta Method. This is intended to be used directly with the Power Analysis so the user is expected to use CUPAC. This simplifies a bit the usage although it is a slight approximation but should work for our implementation. Otherwise we have to change how the interface works so we can pass ratio_metrics and normal metrics, which might imply a ton of refactoring and changing how users usually interact with the library.
Delta Method Isolated would be doing the Delta Method Properly with CUPAC, were we don't predict the ratio but both numerator and denominator and then apply the Delta Method on that to consider it when using CUPED. As can be seen predicting the ratio is a good approximate and tends to overestimate variance. Recall that we apply Delta Method as otherwise we underestimate variance. I prefer having a very small type I error rather than changing how users interact.
Please feel free to check it and we can discuss implementation and align on the best way going forward!
:warning: Please install the to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
:x: Patch coverage is 97.56098% with 3 lines in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 96.57%. Comparing base (9216f8f) to head (607314b).
| Files with missing lines | Patch % | Lines |
|---|---|---|
| cluster_experiments/power_analysis.py | 84.61% | 2 Missing :warning: |
| cluster_experiments/experiment_analysis.py | 98.97% | 1 Missing :warning: |
| :exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality. |
Additional details and impacted files
@@ Coverage Diff @@
## main #230 +/- ##
==========================================
+ Coverage 96.30% 96.57% +0.27%
==========================================
Files 17 17
Lines 1679 1783 +104
==========================================
+ Hits 1617 1722 +105
+ Misses 62 61 -1
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
Codecov is not updating. The only line that is not covered is present in CUPAC. Not sure if it is really needed, seems a bit too much IMO, but we could add a test in power config to test that CUPAC with Delta Method is indeed working.
I'm going to write these missing tests:
- [x] PowerAnalysis with delta method, from config and with covariates
- [x] AnalysisPlan with delta method, from config and with covariates
- [x] PowerAnalysis with delta method, with cupac
- [x] PowerAnalysis with delta method breaks with non-aggregated data
- [x] DeltaMethod and clustered ols give the same results with covariates
Also a small notebook showing how delta method with cupac works
Missing:
- [x] Clean delta method code
- [x] Add tests with multiple covariates
- [x] Understand why delivery time doesn't work in notebook
- [x] Add documentation of how exactly we're using delta method with covariate adjustment
- [x] Stricter tests on delta method estimator (point estimate + variance, are we doing it well?)