expan
expan copied to clipboard
Fix early stopping, by making it stricter
The implementation of "alpha spending" increased our Type I error rate from 5% to approximately 10%, as it didn't spend alpha 'piece-wise' but instead it spent it 'cumulatively'
Therefore, this PR makes the early stopping stricter by an arbitrary factor of 100. As a result, we will get an 'extra Type I error rate' of approximately 0.05% instead of 5%.
Then, the total Type I error rate will be 5.05% (= 5%+0.05%) instead of 10% (=5%+5%)
Coverage increased (+0.03%) to 92.412% when pulling a6e2523fad44aad6c57b72f12a7be9714030f472 on fix-early-stopping.Solution2 into 5fa3dd1a0833530fe6bf659544e98a0b9fe9f9b9 on master.
In practice though, we are unlikely to use this internally without further investigation of the implications. In the last few minutes we had a meeting about this option, and other options, for proceeding
Therefore, we won't immediately merge this