Is there an empirically optimal maxVisits value that can yield more accurate winrate data?

[qcgm1978]

I would like to know if there is an empirically optimal maxVisits value that can yield more accurate winrate data. Setting a higher value requires more time, so I’m looking for a more economical way to obtain winrate fluctuation data across around 100 games. Are there any relevant studies on this? My current configuration is as follows:

path: "/opt/homebrew/Cellar/katago/1.13.2/bin/katago"

arguments: "analysis 
-model 
/opt/homebrew/Cellar/katago/1.13.2/share/katago/g170-b30c320x2-s4824661760-d1229536699.bin.gz 
-config 
/opt/homebrew/Cellar/katago/1.13.2/share/katago/configs/analysis_example.cfg"

maxVisits: 16

[OmnipotentEntity]

You can create some sort of metric by which to measure the time/accuracy tradeoff. In engineering, this is called constructing a "figure of merit." After constructing this value, you may then optimize according to that and arrive at an "empirically optimal" value for this parameter. However, because this is an inherently 2 dimensional concept, depending on how you weight these values you'll get different results for your optimization. Your value will only necessarily be the best value for that particular figure of merit, not for any possible figure of merit you could construct.

[qcgm1978]

Thank you, I will look into the “figure of merit” you mentioned. I appreciate you providing this line of thinking - I will try to find a good solution, especially some kind of benchmark to validate the winrates.

[lightvector]

I would also recommend using a newer neural net. "g170-b30c320x2-s4824661760-d1229536699.bin.gz " is very old and likely much worse than the current 18b net at https://katagotraining.org/networks/ so if you want better analysis, you can probably get an improvement for free just by upgrading.

[qcgm1978]

Thanks. I've just downloaded the highest Elo Rating Network File in the page.

[kaorahi]

to get a rough idea...

[qcgm1978]

Wonderful. It seems that the results would be acceptable when maxVisits==1000. I'll try it in my code.

[qcgm1978]

I have another problem now. Which is more important for the accuracy of the win rate, maxVariationsForEachMove or maxVisits, and how to balance them to achieve higher efficiency?

[qcgm1978]

Google bard answer me as follows:

If you are playing in a tournament or other competitive setting, you should prioritize accuracy over efficiency. In this case, you should set maxVariationsForEachMove to a high value, such as 1000.

If you are playing for fun or practicing, you can prioritize efficiency over accuracy. In this case, you can reduce maxVariationsForEachMove to a lower value, such as 100 or 10.

If you are playing on a computer with limited resources, you will need to reduce both maxVariationsForEachMove and maxVisits to improve efficiency.

[lightvector]

"maxVariationsForEachMove" is not a parameter in KataGo, and I don't think the values given to you by bard are likely to have any meaningful use (bard is probably hallucinating and making up fake recommendations about something it doesn't understand, as LLMs tend to do). You should set maxVisits as high as you can tolerate given your compute or time budget, and as far as I know there isn't any other parameter that it should be balanced against.

[qcgm1978]

@lightvector You're right. I misunderstood the option in analyze-sgf. In that js module maxVariationsForEachMove have no effect on the accuracy. KataGo generates so many variations for a move, and the maximun variations count for a move of a SGF with the option.

[qcgm1978]

Based on my recent experiments, I've found that the optimal solution is likely to be found when the number of visits reaches around 2000.