denseReg icon indicating copy to clipboard operation
denseReg copied to clipboard

AssertionError: Model diverged with loss = NaN

Open konrad-marzec opened this issue 5 years ago • 1 comments

python model/hourglass_um_crop_tiny.py --dataset 'msra' --pid 0 --batch_size 3 --num_stack 2 --num_fea 128 --debug_level 2 --is_train False

finally into the long long training loop
Traceback (most recent call last):
  File "model/hourglass_um_crop_tiny.py", line 932, in <module>
    run_train(dataset, val_dataset)
  File "model/hourglass_um_crop_tiny.py", line 889, in run_train
    train(model, restore_step)
  File "/home/denseReg/model/train_single_gpu.py", line 149, in train
    assert not np.isnan(loss_value), 'Model diverged with loss = NaN'
AssertionError: Model diverged with loss = NaN

konrad-marzec avatar May 11 '19 11:05 konrad-marzec

@melonwan can you help ?

konrad-marzec avatar Jun 20 '19 10:06 konrad-marzec