vowpal_wabbit
vowpal_wabbit copied to clipboard
VowpalWabbit
[wip] experimenting with tests for bad squarecb predictions issue
Just wanted to ask your opinion on doing things like that. The goal is to write tests that are asserting specific invariants vs checking stdout that we have now - looks like we really need to have some tool for that. For example here we can check if every of many different exploration strategies (adding more is almost config change) produces valid probabilities. Cons here:
- Dependency on vw_executor is introduced which is probably not great. We can replace it with just pyvw, but the code will become a bit more verbose.
- It would be better to replace checked in datafiles like the one i have here with the code that generates them - happy to do it if we decide to proceed.
In general it is just a quick draft illustrating approach - happy to hear opinions whether we should continue moving in this direction and then polish details. Also happy to hear about other ways of writing similar tests (in case I miss something)
Just wanted to ask your opinion on doing things like that. The goal is to write tests that are asserting specific invariants vs checking stdout that we have now - looks like we really need to have some tool for that. For example here we can check if every of many different exploration strategies (adding more is almost config change) produces valid probabilities. Cons here:
1. Dependency on vw_executor is introduced which is probably not great. We can replace it with just pyvw, but the code will become a bit more verbose. 2. It would be better to replace checked in datafiles like the one i have here with the code that generates them - happy to do it if we decide to proceed.In general it is just a quick draft illustrating approach - happy to hear opinions whether we should continue moving in this direction and then polish details. Also happy to hear about other ways of writing similar tests (in case I miss something)
There's some benefit in implementing some of these as a test-debug reduction and that way that could be turned on to debug at any point. The bad thing is that they would have to be implemented in cpp.
