fishtest bias by test time?

[vondele]

This is more a question, rather than an issue. Probably for @vdbergh .

Could it be that our fishtest SPRT results are biased due to the fact that our SPRT is not really sequential, and that the results return in an order that depends on the runtime of a test?

For example, I seem to notice that the first results returned during local testing are almost always draws, e.g. quick 3-fold repetitions. Similarly, imagine a patch that makes an engine win more often, but needing 1000move games. These results would take a long time to arrive, and especially with many workers, the master, that has its wins in short games, would initially be reporting more wins than losses.

Just food for thought, nothing urgent IMO.

[ppigazzini]

Similarly, imagine a patch that makes an engine win more often, but needing 1000move games. These results would take a long time to arrive, and especially with many workers, the master, that has its wins in short games, would initially be reporting more wins than losses.

With pentanomial frequencies Master and Patch should have the same probability to get short {D+D} and short {W+L}, Patch should have a higher probability to get long {W+D}s. If these long {W+D}s are uniformly distributed in the opening batch of the workers, we should have the risk to stop the SPRT (because Master and Patch are even) before the analysis of the long {W+D}s, but we should have an increasing percentage (wrt the time) of workers "stuck" playing the long {W+D}s.