models
models copied to clipboard
[Task] Update evaluation in retrieval model examples
Description
Currently, 05-Retrieval-Model.ipynb doesnt evaluate the model (model.evaluate).
I think we should extend the example by model.evaluate.
- We should add a description, when using a validation dataset in the model.fit, that the validation scores depending on the validation batch size, as it uses negative samples.
- We should use model.evaluation (once without item corpus and once with item corpus) and explaining the difference + explaining the performance.
- We should add local prediction to the example. How to generate topk recommendations per user and save it to a file.
@sararb @gabrielspmoreira related to 2. -> how does the evaluation with itemcorpus work, when a user has multiple positive interactions? E.g. the users watched movies and watched ~5 movies. Do we evaluate each positive example against the full item corpus and treat everything negative? So we predict the full item corpus 5 times and average the metrics?
@rnyak Can you add your comments/requirements, as well?