how to evaluate a model?

[zhifeng-huang]

I have followed the tutorial to implement a model for my own dataset. However, I have no idea how good the performance it is.

I am turning hyperparameter.

What does it mean if I get the following result? Top-100 accuracy (train): 0.28. Top-100 accuracy (test): 0.09.

What does it mean if I get the following result? Top-100 accuracy (train): 0.22. Top-100 accuracy (test): 0.22.

I trained the model so the loss did not go down anymore. I still could not bring up the accuracy. Any suggestion what I can do?

[ageofneil]

Your first result indicate the model is overfitting. This means it wont generalize well. It's memorizing the training data. Your second model has a lower accuracy than the first, but it generalizes well to new data.

I'de recommend reading "Examples of Bias and Variance" by Andrew Ng for a basic overview of what these numbers mean.

[dgoldenberg-audiomack]

I think the doc would benefit from a specific how-to on how to tell if it's overfitting and what to try to combat the issue.

[almirb]

I think the doc would benefit from a specific how-to on how to tell if it's overfitting and what to try to combat the issue.

I think this would clarify what could be done a little: https://www.tensorflow.org/tutorials/keras/overfit_and_underfit

[dgoldenberg-audiomack]

@almirb Yeah, nice but won't this take months or years to get all done and right? I think there are mlops frameworks out there to automate this kind of thing, otherwise it's an endless trial-and-error loop...

[almirb]

@almirb Yeah, nice but won't this take months or years to get all done and right? I think there are mlops frameworks out there to automate this kind of thing, otherwise it's an endless trial-and-error loop...

Yes.. I am also getting a hard time trying to get all those things to work and create a production grade recsys. TFRS is a great framework! But I'm new to keras and tensorflow. A lot of things to learn...