vowpal_wabbit icon indicating copy to clipboard operation
vowpal_wabbit copied to clipboard

Add automated tests for checking correct state preservation in models

Open olgavrou opened this issue 3 years ago • 2 comments

Short description

We keep getting bug reports (usually coming from python) where the predictions between a trained vw instance differ if you save the model and then load it (without any training in between). i.e.:

predictions from:

  1. initialize vw instance
  2. train
  3. predict

differs from

  1. initialize vw instance
  2. train
  3. save model (not referring to save_resume here)
  4. load model
  5. predict

Possible solution/implementation details

This could be automated using the python runtests infra to (for each test in RunTests):

  1. create vw instance from a run_test test
  2. train it
  3. predict -> keep predictions 1
  4. save
  5. load
  6. predict -> keep predictions 2
  7. compare predictions 1 with predictions 2

olgavrou avatar Jul 13 '21 16:07 olgavrou

similar to flatbuffer automated tests https://github.com/VowpalWabbit/vowpal_wabbit/blob/2d394ce7bdb02cc3f5bb08c0031565e0e4ea48dc/test/run_tests.py#L19

lalo avatar Jul 13 '21 16:07 lalo

@olgavrou Hi, I would like to try this issue if it is not done yet.

michiboo avatar Feb 18 '23 16:02 michiboo