expan icon indicating copy to clipboard operation
expan copied to clipboard

Fix early stopping, by making it stricter

Open aaron-mcdaid-zalando opened this issue 6 years ago • 2 comments

The implementation of "alpha spending" increased our Type I error rate from 5% to approximately 10%, as it didn't spend alpha 'piece-wise' but instead it spent it 'cumulatively'

Therefore, this PR makes the early stopping stricter by an arbitrary factor of 100. As a result, we will get an 'extra Type I error rate' of approximately 0.05% instead of 5%.

Then, the total Type I error rate will be 5.05% (= 5%+0.05%) instead of 10% (=5%+5%)

aaron-mcdaid-zalando avatar Nov 26 '18 18:11 aaron-mcdaid-zalando

Coverage Status

Coverage increased (+0.03%) to 92.412% when pulling a6e2523fad44aad6c57b72f12a7be9714030f472 on fix-early-stopping.Solution2 into 5fa3dd1a0833530fe6bf659544e98a0b9fe9f9b9 on master.

coveralls avatar Nov 26 '18 18:11 coveralls

In practice though, we are unlikely to use this internally without further investigation of the implications. In the last few minutes we had a meeting about this option, and other options, for proceeding

Therefore, we won't immediately merge this

aaron-mcdaid-zalando avatar Nov 27 '18 14:11 aaron-mcdaid-zalando