[Feature request] validation matches

[Eugenio-Bruno]

To understand how well the tuning process is proceeding, it might be very useful to have periodic validation matches after N iterations (with a typical N of maybe 100). These validation matches would run G games, many more than a typical iteration, maybe even G = N times as much.

This would probably be from a 10% to a 50% slowdown depending on the settings of N and G, but it would show the progress of the tuning.

You make additional graphs with one data point per validation match. When the elo and error bar in the graphs stop moving around, tuning is (probably) done.

I think it's possible for me to do a proof of concept using the data chess-tuning-tools already generates without changing chess-tuning-tools code at all, if you want a proof of concept to better understand what I'm talking about (or if it'd work decently)... If you'd like, maybe you could give me an example tune file for stockfish or something and I could show you the code and graphs from the proof of concept, so that we can exclude the idea not working because my tune file is completely idiotic :P

I'd love to hear your opinion on this :)

[kiudee]

I think the idea is definitely interesting and could help users decide on when to stop tuning as soon as there are diminishing returns. Implementation-wise this is straightforward to implement and the existing https://github.com/kiudee/chess-tuning-tools/blob/8706ac2d73286cde885582f922223d6a29e9412a/tune/local.py#L189 can be reused. That being said, I don’t know yet when I would be able to work on this, since there are a few things which need to be improved in chess-tuning-tools first.

[Eugenio-Bruno]

I see that hacking it in (not properly) might not be too hard. I'll make a not-for-merge PR just as a prototype.

[Eugenio-Bruno]

PR https://github.com/kiudee/chess-tuning-tools/pull/109

suggestions for test tunes appreciated :)