icme2019-bytedance-grand-challenge
icme2019-bytedance-grand-challenge copied to clipboard
ERROR:tensorflow:Model diverged with loss = NaN.
I0410 22:34:05.116471 4483055104 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0410 22:34:05.160604 4483055104 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Saving checkpoints for 0 into finish/model.ckpt. I0410 22:34:06.100261 4483055104 basic_session_run_hooks.py:606] Saving checkpoints for 0 into finish/model.ckpt. ERROR:tensorflow:Model diverged with loss = NaN. E0410 22:34:13.729731 4483055104 basic_session_run_hooks.py:760] Model diverged with loss = NaN. tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training. 在运行run_model.sh时报错,请问可能的问题是什么?前面的数据处理部分看起来是正常的,尝试过:降低学习率,调高batch_size,换tensorflow版本,都没有用。
same problem with you,do you solve it now?