shorttermScoreError is always -1

[stevenshen121200]

The shorttermScoreError is always -1 when running kata-raw-nn in GTP. So is shorttermWinlossError. Is this a bug or I should modify some corresponding config?

[kaorahi]

You may need a newer model? I get -1.000 and 0.820 with g170-b10c128-s1141046784-d204142634 and kata1-b40c256-s11101799168-d2715431527, respectively.

[lightvector]

Sorry, yeah old models do not support this output. Right now a default value of -1 is filled in in such cases, probably we should document that.

[stevenshen121200]

Thank you for answering me. If available, I want to know the smallest model that supports the shorttermScoreError output. Also, I'm wondering if this value is related to the accuracy of the predicted score.

[lightvector]

There isn't any model smaller than 40b that supports the output - it was introduced around the time when distributed training started.

It does NOT measure the accuracy or the error of the predicted score. It measures the expected short term variation in the predicted score. For example, on the first move of the game:

• The predicted score has very high average error, because the entire game is remaining and there is the maximum amount of time left for variance in the score to occur.
• The predicted score has very low expected short term variation, because the next few moves in the game are likely to be standard corner moves and these moves are not likely to change the evaluation of the score a lot.

You could of course theorize that maybe this value is related to the average error with respect to the game-theoretic-optimal score, but that is hard to measure because we don't in general know what optimal play is. :)