models
models copied to clipboard
[Task] Add regression tests for session-based models
Description
We want to add a CI test, that collects metrics to understand if performance changes between releases.
First step:
- Add regression tests for session based transformer model https://github.com/NVIDIA-Merlin/models/blob/main/examples/usecases/transformers-next-item-prediction.ipynb
@radekosmulski shared ideas on it here: https://nvidia.slack.com/archives/CVBDJUPEZ/p1679617586380369
We will collect:
- Runtime NVTabular
- Runtime Training Model with TensorFlow
- NDCG Performance of Session-based model
I close other tickets related to it:
- https://github.com/NVIDIA-Merlin/Merlin/issues/223 : I think that is solved based on sample PR)
- https://github.com/NVIDIA-Merlin/Merlin/issues/233 : We changed priority to session-based models