models icon indicating copy to clipboard operation
models copied to clipboard

[Task] Add regression tests for session-based models

Open bschifferer opened this issue 2 years ago • 0 comments

Description

We want to add a CI test, that collects metrics to understand if performance changes between releases.

First step:

  • Add regression tests for session based transformer model https://github.com/NVIDIA-Merlin/models/blob/main/examples/usecases/transformers-next-item-prediction.ipynb

@radekosmulski shared ideas on it here: https://nvidia.slack.com/archives/CVBDJUPEZ/p1679617586380369

We will collect:

  • Runtime NVTabular
  • Runtime Training Model with TensorFlow
  • NDCG Performance of Session-based model

I close other tickets related to it:

  • https://github.com/NVIDIA-Merlin/Merlin/issues/223 : I think that is solved based on sample PR)
  • https://github.com/NVIDIA-Merlin/Merlin/issues/233 : We changed priority to session-based models

bschifferer avatar Apr 03 '23 11:04 bschifferer