pykan icon indicating copy to clipboard operation
pykan copied to clipboard

Loss suddenly become Nan during training process.

Open mosimod opened this issue 9 months ago • 1 comments

train loss: 5.15e-01 | test loss: 5.33e-01 | reg: 1.47e+01 : 44%|▍| 436/1000 [04:19<06:55, 1.36it/ train loss: 5.15e-01 | test loss: 5.33e-01 | reg: 1.47e+01 : 44%|▍| 437/1000 [04:19<07:05, 1.32it/ train loss: 5.17e-01 | test loss: 5.34e-01 | reg: 1.47e+01 : 44%|▍| 437/1000 [04:20<07:05, 1.32it/ train loss: 5.17e-01 | test loss: 5.34e-01 | reg: 1.47e+01 : 44%|▍| 438/1000 [04:20<07:14, 1.29it/ train loss: 5.11e-01 | test loss: 5.36e-01 | reg: 1.47e+01 : 44%|▍| 438/1000 [04:20<07:14, 1.29it/ train loss: 5.11e-01 | test loss: 5.36e-01 | reg: 1.47e+01 : 44%|▍| 439/1000 [04:20<05:56, 1.57it/ train loss: 5.13e-01 | test loss: 5.27e-01 | reg: 1.47e+01 : 44%|▍| 439/1000 [04:20<05:56, 1.57it/ train loss: 5.13e-01 | test loss: 5.27e-01 | reg: 1.47e+01 : 44%|▍| 440/1000 [04:20<04:59, 1.87it/ train loss: 5.19e-01 | test loss: 5.18e-01 | reg: 1.47e+01 : 44%|▍| 440/1000 [04:21<04:59, 1.87it/ train loss: 5.19e-01 | test loss: 5.18e-01 | reg: 1.47e+01 : 44%|▍| 441/1000 [04:21<04:09, 2.24it/ train loss: 5.13e-01 | test loss: 5.38e-01 | reg: 1.47e+01 : 44%|▍| 441/1000 [04:21<04:09, 2.24it/ train loss: 5.13e-01 | test loss: 5.38e-01 | reg: 1.47e+01 : 44%|▍| 442/1000 [04:21<03:38, 2.55it/ train loss: 5.18e-01 | test loss: 5.34e-01 | reg: 1.46e+01 : 44%|▍| 442/1000 [04:21<03:38, 2.55it/ train loss: 5.18e-01 | test loss: 5.34e-01 | reg: 1.46e+01 : 44%|▍| 443/1000 [04:21<04:10, 2.23it/ train loss: 5.14e-01 | test loss: 5.28e-01 | reg: 1.46e+01 : 44%|▍| 443/1000 [04:22<04:10, 2.23it/ train loss: 5.14e-01 | test loss: 5.28e-01 | reg: 1.46e+01 : 44%|▍| 444/1000 [04:22<04:16, 2.16it/ train loss: 5.13e-01 | test loss: 5.32e-01 | reg: 1.48e+01 : 44%|▍| 444/1000 [04:23<04:16, 2.16it/ train loss: 5.13e-01 | test loss: 5.32e-01 | reg: 1.48e+01 : 44%|▍| 445/1000 [04:23<05:15, 1.76it/ train loss: 5.12e-01 | test loss: 5.32e-01 | reg: 1.47e+01 : 44%|▍| 445/1000 [04:23<05:15, 1.76it/ train loss: 5.12e-01 | test loss: 5.32e-01 | reg: 1.47e+01 : 45%|▍| 446/1000 [04:23<05:39, 1.63it/ train loss: 5.12e-01 | test loss: 5.22e-01 | reg: 1.46e+01 : 45%|▍| 446/1000 [04:24<05:39, 1.63it/ train loss: 5.12e-01 | test loss: 5.22e-01 | reg: 1.46e+01 : 45%|▍| 447/1000 [04:24<06:27, 1.43it/ train loss: 5.13e-01 | test loss: 5.31e-01 | reg: 1.46e+01 : 45%|▍| 447/1000 [04:25<06:27, 1.43it/ train loss: 5.13e-01 | test loss: 5.31e-01 | reg: 1.46e+01 : 45%|▍| 448/1000 [04:25<06:21, 1.45it/ train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 448/1000 [04:26<06:21, 1.45it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 449/1000 [04:26<06:21, 1.44it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 449/1000 [04:26<06:21, 1.44it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 450/1000 [04:26<06:07, 1.50it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 450/1000 [04:27<06:07, 1.50it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 451/1000 [04:27<06:07, 1.50it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 451/1000 [04:28<06:07, 1.50it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 452/1000 [04:28<06:01, 1.51it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 452/1000 [04:28<06:01, 1.51it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 453/1000 [04:28<06:00, 1.52it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 453/1000 [04:29<06:00, 1.52it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 454/1000 [04:29<05:57, 1.53it/s] train loss: nan | test loss: nan | reg: nan : 45%|██████▎ | 454/1000 [04:30<05:57, 1.53it/s] train loss: nan | test loss: nan | reg: nan : 46%|██████▎ | 455/1000 [04:30<05:58, 1.52it/s] train loss: nan | test loss: nan | reg: nan : 46%|██████▎ | 455/1000 [04:30<05:58, 1.52it/s] train loss: nan | test loss: nan | reg: nan : 46%|██████▍ | 456/1000 [04:30<05:59, 1.51it/s] train loss: nan | test loss: nan | reg: nan : 46%|██████▍ | 456/1000 [04:31<05:59, 1.51it/s] train loss: nan | test loss: nan | reg: nan : 46%|██████▍ | 457/1000 [04:31<06:26, 1.41it/s] train loss: nan | test loss: nan | reg: nan : 46%|██████▍ | 457/1000 [04:32<06:26, 1.41it/s] train loss: nan | test loss: nan | reg: nan : 46%|██████▍ | 458/1000 [04:32<06:20, 1.42it/s] train loss: nan | test loss: nan | reg: nan : 46%|██████▍ | 458/1000 [04:32<06:20, 1.42it/s] train loss: nan | test loss: nan | reg: nan : 46%|██████▍ | 459/1000 [04:32<06:08, 1.47it/s]

mosimod avatar May 20 '24 04:05 mosimod