cluster-experiments icon indicating copy to clipboard operation
cluster-experiments copied to clipboard

feat: Add Variance Reduction to Delta Method and couple it with CUPAC

Open LGonzalezGomez opened this issue 9 months ago • 2 comments

CUPED is used with the Delta Method. This is intended to be used directly with the Power Analysis so the user is expected to use CUPAC. This simplifies a bit the usage although it is a slight approximation but should work for our implementation. Otherwise we have to change how the interface works so we can pass ratio_metrics and normal metrics, which might imply a ton of refactoring and changing how users usually interact with the library.

Delta Method Isolated would be doing the Delta Method Properly with CUPAC, were we don't predict the ratio but both numerator and denominator and then apply the Delta Method on that to consider it when using CUPED. As can be seen predicting the ratio is a good approximate and tends to overestimate variance. Recall that we apply Delta Method as otherwise we underestimate variance. I prefer having a very small type I error rather than changing how users interact.

image image Screenshot 2025-03-02 at 20 00 08

Please feel free to check it and we can discuss implementation and align on the best way going forward!

LGonzalezGomez avatar Mar 02 '25 19:03 LGonzalezGomez

:warning: Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

:x: Patch coverage is 97.56098% with 3 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 96.57%. Comparing base (9216f8f) to head (607314b).

Files with missing lines Patch % Lines
cluster_experiments/power_analysis.py 84.61% 2 Missing :warning:
cluster_experiments/experiment_analysis.py 98.97% 1 Missing :warning:
:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #230      +/-   ##
==========================================
+ Coverage   96.30%   96.57%   +0.27%     
==========================================
  Files          17       17              
  Lines        1679     1783     +104     
==========================================
+ Hits         1617     1722     +105     
+ Misses         62       61       -1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov-commenter avatar Mar 02 '25 19:03 codecov-commenter

Codecov is not updating. The only line that is not covered is present in CUPAC. Not sure if it is really needed, seems a bit too much IMO, but we could add a test in power config to test that CUPAC with Delta Method is indeed working.

LGonzalezGomez avatar Mar 06 '25 07:03 LGonzalezGomez

I'm going to write these missing tests:

  • [x] PowerAnalysis with delta method, from config and with covariates
  • [x] AnalysisPlan with delta method, from config and with covariates
  • [x] PowerAnalysis with delta method, with cupac
  • [x] PowerAnalysis with delta method breaks with non-aggregated data
  • [x] DeltaMethod and clustered ols give the same results with covariates

Also a small notebook showing how delta method with cupac works

david26694 avatar Aug 01 '25 09:08 david26694

Missing:

  • [x] Clean delta method code
  • [x] Add tests with multiple covariates
  • [x] Understand why delivery time doesn't work in notebook
  • [x] Add documentation of how exactly we're using delta method with covariate adjustment
  • [x] Stricter tests on delta method estimator (point estimate + variance, are we doing it well?)

david26694 avatar Aug 06 '25 15:08 david26694