Pytorch-UNet
Pytorch-UNet copied to clipboard
milesial
get error when train
when the epoch 1 run for a time the error was raised: 'ValueError: autodetected range of [nan, nan] is not finite' do u know how to fix this problem?Thanks for ur time
wandb: Run summary: wandb: epoch 0 wandb: step 458 wandb: train loss nan when the error happen wandb show this message
Me too, did someone sove the problem?
in my case, the problem occurs in epoch 18
ValueError: autodetected range of [-0.4384765625, inf] is not finite
INFO: Validation Dice score: 0.9037662744522095
Epoch 19/300: 18%|████████████▊ | 24/135 [00:09<00:42, 2.60img/s, loss (batch)=0.34]
Traceback (most recent call last):
File ".../Pytorch-UNet/train.py", line 207, in <module>
train_net(net=net,
File "../Pytorch-UNet/train.py", line 141, in train_net
histograms['Gradients/' + tag] = wandb.Histogram(value.grad.data.cpu())
File "../anaconda3/envs/pytorch-unet/lib/python3.9/site-packages/wandb/sdk/data_types/histogram.py", line 77, in __init__
self.histogram, self.bins = np.histogram(sequence, bins=num_bins)
File "<__array_function__ internals>", line 180, in histogram
File "../anaconda3/envs/pytorch-unet/lib/python3.9/site-packages/numpy/lib/histograms.py", line 793, in histogram
bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
File "../anaconda3/envs/pytorch-unet/lib/python3.9/site-packages/numpy/lib/histograms.py", line 426, in _get_bin_edges
first_edge, last_edge = _get_outer_edges(a, range)
File "../anaconda3/envs/pytorch-unet/lib/python3.9/site-packages/numpy/lib/histograms.py", line 323, in _get_outer_edges
raise ValueError(
ValueError: autodetected range of [-0.4384765625, inf] is not finite
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb: / 41.861 MB of 41.861 MB uploaded (0.000 MB deduped)
wandb: Run history:
