recommenders icon indicating copy to clipboard operation
recommenders copied to clipboard

loss becomes nan

Open zhifeng-huang opened this issue 2 years ago • 1 comments

58/108 [===============>..............] - ETA: 1:05:49 - root_mean_squared_error: nan - factorized_top_k/top_1_categorical_accuracy: 0.0045 - factorized_top_k/top_5_categorical_accuracy: 0.0089 - factorized_top_k/top_10_categorical_accuracy: 0.0130 - factorized_top_k/top_50_categorical_accuracy: 0.0295 - factorized_top_k/top_100_categorical_accuracy: 0.0380 - loss: nan - regularization_loss: nan - total_loss: nan

why do we have the loss becoming nan? Any tips appreciated.

Thanks, Robert

zhifeng-huang avatar Mar 13 '23 22:03 zhifeng-huang

I believe I ran to these when I have some NAN values in the input or improperly scaled values. There are other reasons too. Check this SO.

https://stackoverflow.com/questions/40050397/deep-learning-nan-loss-reasons

But the easiest things to check are NAN values, and improperly scaled values.

krsnewwave avatar Mar 15 '23 16:03 krsnewwave