models icon indicating copy to clipboard operation
models copied to clipboard

[Task] Update evaluation in retrieval model examples

Open bschifferer opened this issue 3 years ago • 1 comments

Description

Currently, 05-Retrieval-Model.ipynb doesnt evaluate the model (model.evaluate).

I think we should extend the example by model.evaluate.

  1. We should add a description, when using a validation dataset in the model.fit, that the validation scores depending on the validation batch size, as it uses negative samples.
  2. We should use model.evaluation (once without item corpus and once with item corpus) and explaining the difference + explaining the performance.
  3. We should add local prediction to the example. How to generate topk recommendations per user and save it to a file.

@sararb @gabrielspmoreira related to 2. -> how does the evaluation with itemcorpus work, when a user has multiple positive interactions? E.g. the users watched movies and watched ~5 movies. Do we evaluate each positive example against the full item corpus and treat everything negative? So we predict the full item corpus 5 times and average the metrics?

bschifferer avatar Aug 23 '22 08:08 bschifferer

@rnyak Can you add your comments/requirements, as well?

bschifferer avatar Aug 23 '22 08:08 bschifferer