denseReg
denseReg copied to clipboard
AssertionError: Model diverged with loss = NaN
python model/hourglass_um_crop_tiny.py --dataset 'msra' --pid 0 --batch_size 3 --num_stack 2 --num_fea 128 --debug_level 2 --is_train False
finally into the long long training loop
Traceback (most recent call last):
File "model/hourglass_um_crop_tiny.py", line 932, in <module>
run_train(dataset, val_dataset)
File "model/hourglass_um_crop_tiny.py", line 889, in run_train
train(model, restore_step)
File "/home/denseReg/model/train_single_gpu.py", line 149, in train
assert not np.isnan(loss_value), 'Model diverged with loss = NaN'
AssertionError: Model diverged with loss = NaN
@melonwan can you help ?