fil_backend icon indicating copy to clipboard operation
fil_backend copied to clipboard

Published 20 hours ago • verified publisher icon triton-inference-server

Rewrite tests

Open RAMitchell opened this issue 2 years ago • 1 comments

This PR moves the model generation and loading into pytest.

This achieves a few things:

  • Generation of configurations and models on the fly by pytest
  • Testing prediction results directly against original libraries instead of gtil
  • The ability to test shapley values
  • Less code
  • Fixes #348

The tests run faster than before, are easy to extend and we can generate more coverage.

Todo:

  • [x] cuml tests
  • [x] sklearn tests
  • [x] update docs

RAMitchell avatar Mar 22 '23 15:03 RAMitchell

I have updated this so that models are generated using the original code but cached automatically using joblib.memory. Here below is an example of the test matrix (68 items), taking 130s. I think it will be particularly useful to test the experimental_optimization flag for all as the backend is updated with the latest cuml changes.

../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_xgb[instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_xgb[instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_xgb[instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_xgb[instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_lgbm[instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_lgbm[instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_lgbm[instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_lgbm[instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn[instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn[instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn[instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn[instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_cuml[instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_cuml[instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_cuml[instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_cuml[instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn_gbm[instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn_gbm[instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn_gbm[instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn_gbm[instance:KIND_GPU-exper_optim:False] PASSED
```

RAMitchell avatar Apr 03 '23 14:04 RAMitchell