Context-Transformer icon indicating copy to clipboard operation
Context-Transformer copied to clipboard

Pretraining RFBNet on source domain dataset COCO60, it shows loss nan

Open LexieYang opened this issue 3 years ago • 7 comments

Hello, I really appreciate your work! When I pretrain RFBNet on source domain dataset COCO60, it shows loss nan. Is it correct and why?

Screen Shot 2021-04-25 at 10 24 49 PM

LexieYang avatar Apr 26 '21 02:04 LexieYang

I have the same problem from iter2940...Is there anybody know why?

whattoshow avatar Jun 09 '21 03:06 whattoshow

Hello ,I've asked my classmate, they told me the exploding gradient may cause this problem.

whattoshow avatar Jun 09 '21 07:06 whattoshow

Hi, do you make any modifications to the code? If so, please first try with the original code.

Ze-Yang avatar Jun 09 '21 08:06 Ze-Yang

Hi, do you make any modifications to the code? If so, please first try with the original code.

Thanks a lot for your reply~!!! Due to my computer's GPU memory is small , I've got 'CUDA out of memory ' error when running the original code,so I changed the batch size to 32, then pretrain RFBNet on source domain dataset COCO60, and met the same problem of LexieYang's.

whattoshow avatar Jun 10 '21 07:06 whattoshow

I will verify it and get back to you soon.

Ze-Yang avatar Jun 10 '21 07:06 Ze-Yang

I will verify it and get back to you soon.

Thank you!!!!:blush:

whattoshow avatar Jun 10 '21 07:06 whattoshow

Hello! I am encountering the same problem, have there been any updates regarding this?

jamesrosstwo avatar Jul 05 '22 18:07 jamesrosstwo