fishtest icon indicating copy to clipboard operation
fishtest copied to clipboard

SPRT test not accepted with large LLR

Open vondele opened this issue 3 years ago • 10 comments

https://tests.stockfishchess.org/html/live_elo.html?61a9d4bf9e8855bba1a35c4f

was stopped by hand after reaching 3.11

maybe related to the fact that there are many purged workers ?

vondele avatar Dec 05 '21 12:12 vondele

Weird, green residual purged too. Do we know if this was purged by hand?

ppigazzini avatar Dec 05 '21 13:12 ppigazzini

This test had auto purge on. So perhaps it just kept on purging....

https://tests.stockfishchess.org/actions?count=1&before=1638681200.308

vdbergh avatar Dec 05 '21 16:12 vdbergh

The fact that green workers were purged is weird. Green workers can in principle be purged if the p value is very low. But if all workers are green then you would not expect the p value to be very low.

vdbergh avatar Dec 05 '21 16:12 vdbergh

The fact that green workers were purged is weird. Green workers can in principle be purged if the p value is very low. But if all workers are green then you would not expect the p value to be very low.

Ah but that's precisely what's happening. The p-value is still 0.00 despite all the workers now being green. Unfortunately it is tricky right now to audit the chi2 computation.

There should be a summary table per user, with an option of csv export.

vdbergh avatar Dec 05 '21 16:12 vdbergh

It might be interesting to know how much more purging would be necessary for the p-value to go up.

vdbergh avatar Dec 05 '21 17:12 vdbergh

you want me to push purge more on the test?

vondele avatar Dec 05 '21 17:12 vondele

We could. But perhaps @ppigazzini can transfer the db to DEV?

vdbergh avatar Dec 05 '21 18:12 vdbergh

let me purge now a couple of times, doesn't really make a difference

vondele avatar Dec 05 '21 18:12 vondele

btw, we should make you an approver on fishtest..

vondele avatar Dec 05 '21 18:12 vondele

I pressed purge once more (IIRC), and the run got LLR of 2.71 and started running again. p value 0.01%

vondele avatar Dec 05 '21 18:12 vondele