vowpal_wabbit
vowpal_wabbit copied to clipboard
Add automated tests for checking correct state preservation in models
Short description
We keep getting bug reports (usually coming from python) where the predictions between a trained vw instance differ if you save the model and then load it (without any training in between). i.e.:
predictions from:
- initialize vw instance
- train
- predict
differs from
- initialize vw instance
- train
- save model (not referring to save_resume here)
- load model
- predict
Possible solution/implementation details
This could be automated using the python runtests infra to (for each test in RunTests):
- create vw instance from a run_test test
- train it
- predict -> keep predictions 1
- save
- load
- predict -> keep predictions 2
- compare predictions 1 with predictions 2
similar to flatbuffer automated tests https://github.com/VowpalWabbit/vowpal_wabbit/blob/2d394ce7bdb02cc3f5bb08c0031565e0e4ea48dc/test/run_tests.py#L19
@olgavrou Hi, I would like to try this issue if it is not done yet.