icme2019-bytedance-grand-challenge icon indicating copy to clipboard operation
icme2019-bytedance-grand-challenge copied to clipboard

ERROR:tensorflow:Model diverged with loss = NaN.

Open FantaXuan opened this issue 2 years ago • 1 comments

I0410 22:34:05.116471 4483055104 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0410 22:34:05.160604 4483055104 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Saving checkpoints for 0 into finish/model.ckpt. I0410 22:34:06.100261 4483055104 basic_session_run_hooks.py:606] Saving checkpoints for 0 into finish/model.ckpt. ERROR:tensorflow:Model diverged with loss = NaN. E0410 22:34:13.729731 4483055104 basic_session_run_hooks.py:760] Model diverged with loss = NaN. tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training. 在运行run_model.sh时报错,请问可能的问题是什么?前面的数据处理部分看起来是正常的,尝试过:降低学习率,调高batch_size,换tensorflow版本,都没有用。

FantaXuan avatar Apr 10 '22 14:04 FantaXuan

same problem with you,do you solve it now?

xuanzhiliu avatar Nov 03 '22 06:11 xuanzhiliu