Loss function returns nan

[mostafajahanifar]

System used: Ubuntu 18.04 Tensorflow-gpu 2.1

---

I used the "2_model_SimCR.ipynb" to train a model. After two epochs, the loss function returns "nan" values and messes up the training. I wonder if you have any solution for this?

Train for 53 steps, validate for 12 steps
Epoch 1/5
52/53 [============================>.] - ETA: 4s - loss: 582.9939     
Epoch 00001: val_loss improved from inf to 497.00585, saving model to models/trashnet/SimCLR/SimCLR_05_05_11h_05.h5
53/53 [==============================] - 395s 7s/step - loss: 581.4498 - val_loss: 497.0059
Epoch 2/5
52/53 [============================>.] - ETA: 0s - loss: 421.8934 
Epoch 00002: val_loss improved from 497.00585 to 342.78980, saving model to models/trashnet/SimCLR/SimCLR_05_05_11h_05.h5
53/53 [==============================] - 36s 675ms/step - loss: 420.4594 - val_loss: 342.7898
Epoch 3/5
52/53 [============================>.] - ETA: 0s - loss: 278.3572 
Epoch 00003: val_loss improved from 342.78980 to 213.78286, saving model to models/trashnet/SimCLR/SimCLR_05_05_11h_05.h5
53/53 [==============================] - 37s 694ms/step - loss: 277.1834 - val_loss: 213.7829
Epoch 4/5
52/53 [============================>.] - ETA: 0s - loss: nan      
Epoch 00004: val_loss did not improve from 213.78286
53/53 [==============================] - 34s 643ms/step - loss: nan - val_loss: nan
Epoch 5/5
52/53 [============================>.] - ETA: 0s - loss: nan 
Epoch 00005: val_loss did not improve from 213.78286
53/53 [==============================] - 34s 639ms/step - loss: nan - val_loss: nan
trainable parameters: 11.86 M.
non-trainable parameters: 4.05 M.
Random guess accuracy: 0.0156
accuracy - test - before: 0.74
accuracy - test - after: nan
y_predict_test_before
0.73 | 0.77 | 0.66 | 0.92 | 0.95 | 0.22 | 0.51 | 0.92 | 0.71 | 0.9 | 0.11 | 0.84 | 0.84 | 0.8 | 0.69 |