Seemingly wrong chart in the published versions of the CUPED notebook

[kamicollo]

Hi,

First of all - I found your post on CUPED and comparisons to diff-in-diff and a simple regression with covariates very useful. I'm in a process of "upgrading" how we analyse experiments at my workplace and your work has helped a lot to clarify things.

However - there's one thing that bothered me in your post - the below chart (and associated table just below it):

According to it, the autoregression has the highest variance among all the methods - which I found very counter-intuitive. Surely it would not perform worse that a simple t-test.. The text in the article also suggests otherwise, which made me wonder if there was some strange mistake/issue when the blog post was rendered.

I just cloned your repo and re-ran the notebook, and indeed - I get results that I would expect:

I'm not sure what exactly happened - but it would be great to have those corrected! I'm sure I am not the only one who found your blog posts helpful, and another person may take away the wrong conclusion (that auto-regression is really bad).

[capriceZ]

I was wondering the same when reading the post. I replicated your results. Thanks for sharing your findings!