PaddleOCR icon indicating copy to clipboard operation
PaddleOCR copied to clipboard

Published 20 hours ago • verified publisher icon PaddlePaddle

同时开启AMP和单机多卡训练DBnet++，loss经常Nan

Open hzhuangdy opened this issue 2 years ago • 2 comments

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

  • 系统环境/System Environment：
  • 版本号/Version：Paddle： PaddleOCR： 问题相关组件/Related components：
  • 运行指令/Command Code：
  • 完整报错/Complete Error Message：

hzhuangdy avatar Oct 25 '22 03:10 hzhuangdy

建议先去掉AMP训练

LDOUBLEV avatar Oct 26 '22 03:10 LDOUBLEV

那是不是DBNet++比较不适合使用AMP？我普通训练了20epoch之后用这个模型去做重训开启AMP还是会有同样的问题

hzhuangdy avatar Oct 27 '22 08:10 hzhuangdy

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jul 08 '23 02:07 github-actions[bot]