R2CNN-Plus-Plus_Tensorflow icon indicating copy to clipboard operation
R2CNN-Plus-Plus_Tensorflow copied to clipboard

Published 20 hours ago • verified publisher icon DetectionTeamUCAS

train loss nan

Open congjianting opened this issue 5 years ago • 7 comments

@yangxue0827

你好, 我用自己的训练数据时会出现nan, 我的学习率是0.001, res50网络, 单卡, 这个现象是跟学习率有关吗? 你有什么好的建议解决下吗?

congjianting avatar Aug 19 '19 11:08 congjianting

Mostly there are issues with the dataset.

dexception avatar Sep 23 '19 08:09 dexception

我也出现了nan， 2019-12-03 18:29:49: step210 image_name:b'1114_7186.jpg' | rpn_loc_loss:0.017838023602962494 | rpn_cla_loss:1.3777527809143066 | rpn_total_loss:1.3955907821655273 | fast_rcnn_loc_loss:nan | fast_rcnn_cla_loss:nan | fast_rcnn_loc_rotate_loss:nan | fast_rcnn_cla_rotate_loss:nan | fast_rcnn_total_loss:nan | attention_loss:nan | total_loss:nan | pre_cost_time:0.8750865459442139s

chizhanyuefeng avatar Dec 03 '19 10:12 chizhanyuefeng

请问大家解决问题了吗？

cauivy avatar Dec 17 '19 06:12 cauivy

通常情况下是训练数据存在些问题, 比如越界啥的.

congjianting avatar Dec 17 '19 06:12 congjianting

不过目前用该方法还没有得到一个满意的效果, 暂时换别的方法了.

congjianting avatar Dec 17 '19 06:12 congjianting

There are issues with the dataset.

jamessmith90 avatar Dec 17 '19 08:12 jamessmith90

请问，就这个问题有没有解决办法?

mary-0830 avatar Feb 01 '20 09:02 mary-0830