NaN loss without pretrained model

[Bedrettin-Cetinkaya]

When i trained with pretrained res101 for pascal voc 2007, there is no problem. I could get mAP like yours result's for test data. However, when i try to train pascal voc 2007 without pretrained res101 model, training loss is Nan from start of first epoch. All settings are defaults. Anyone face with/solve this problem?

[adityaarun1]

I believe, an Imagenet pretrained res101/vgg16 model is essential for the network to converge successfully. If you want to train your network from scratch (i.e. using a randomly initialized res101 or vgg16) then first you need to ensure that the intial model parameters are also updated during the training (which is frozen) so that the conv layers actually learn something useful. Also, you may also need to play with various learning rates and other hyperparameter values in order to ensure that the network converges.

[MXLHELLO]

I met the same problem~

[zhengxinvip]

When i trained with pretrained res101 for pascal voc 2007, there is no problem. I could get mAP like yours result's for test data. However, when i try to train pascal voc 2007 without pretrained res101 model, training loss is Nan from start of first epoch. All settings are defaults. Anyone face with/solve this problem?

I meet the same error. have you fix it? thanks.

[zhengxinvip]

I met the same problem~

I meet the same error. have you fix it? thanks.

[MXLHELLO]

Soory..no yet...What about you now？

------------------ 原始邮件 ------------------ 发件人: "tangmiao"[email protected]; 发送时间: 2019年7月26日(星期五) 晚上9:32 收件人: "jwyang/faster-rcnn.pytorch"[email protected]; 抄送: "465586450"[email protected];"Comment"[email protected]; 主题: Re: [jwyang/faster-rcnn.pytorch] NaN loss without pretrained model(#417)

I met the same problem~

I meet the same error. have you fix it? thanks.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[zhengxinvip]

I  have  fix it 。thanks。发自 Smartisan M1L

• body { padding:0 16px 30px !important; margin:0 !important;background-color: #ffffff; line-height: 1.4;word-wrap: break-word; word-break: normal; } div {word-wrap: break-word; word-break: normal; } p {word-wrap: break-word; word-break: normal; text-indent: 0pt !important; } span {word-wrap: break-word; word-break: normal; } a {word-wrap: break-word; word-break: normal; } td {word-wrap: break-word; word-break: break-all; }wonyoung [email protected] 于 2019年8月14日 下午12:01写道：Soory..no yet...What about you now？

------------------ 原始邮件 ------------------

发件人: "tangmiao"[email protected];

发送时间: 2019年7月26日(星期五) 晚上9:32

收件人: "jwyang/faster-rcnn.pytorch"[email protected];

抄送: "465586450"[email protected];"Comment"[email protected];

主题: Re: [jwyang/faster-rcnn.pytorch] NaN loss without pretrained model(#417)

I met the same problem~

I meet the same error. have you fix it? thanks.

—

You are receiving this because you commented.

Reply to this email directly, view it on GitHub, or mute the thread.

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.

[

{

"@context": "http://schema.org",

"@type": "EmailMessage",

"potentialAction": {

"@type": "ViewAction",

"target": "https://github.com/jwyang/faster-rcnn.pytorch/issues/417?email_source=notifications\u0026email_token=ADVZQPC5VOHN53GQUZULN6LQEN7QRA5CNFSM4GOJ4VPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4HUBCA#issuecomment-521093256",

"url": "https://github.com/jwyang/faster-rcnn.pytorch/issues/417?email_source=notifications\u0026email_token=ADVZQPC5VOHN53GQUZULN6LQEN7QRA5CNFSM4GOJ4VPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4HUBCA#issuecomment-521093256",

"name": "View Issue"

},

"description": "View this Issue on GitHub",

"publisher": {

"@type": "Organization",

"name": "GitHub",

"url": "https://github.com"

}

}

] 

[MXLHELLO]

I  have  fix it 。thanks。 You are so capable! So what‘s wrong? Can you tell me how to modify it?

[zhengxinvip]

I have fix it 。thanks。 You are so capable! So what‘s wrong? Can you tell me how to modify it?

you can say other close issue. I fix with it.

[Tom-Cat-God]

You should clip the gradient. add clip_gradient(fasterRCNN, 10.) before optimizer.step()

[countytown]

You should clip the gradient. add clip_gradient(fasterRCNN, 10.) before optimizer.step()

This works for me with res101.yml. The original code only clip_gridient when use vgg16, anyone knows why? Thanks!