FunASR icon indicating copy to clipboard operation
FunASR copied to clipboard

Published 20 hours ago • verified publisher icon modelscope

微调speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch模型训练失败,错误原因There are no improvements in this epoch

Open sunneam opened this issue 1 year ago • 0 comments

[4090c] 2023-11-23 05:36:47,586 (build_trainer:153) INFO: The training was resumed using /home/cnhis/whyme/FunASR/egs_modelscope/asr/TEMPLATE/checkpoint/checkpoint.pb [4090c] 2023-11-23 05:36:47,604 (build_trainer:260) INFO: 4/100epoch started [4090c] 2023-11-23 05:36:50,002 (build_trainer:302) INFO: 4epoch results: [train] time=2.05 seconds, total_count=0, gpu_max_cached_mem_GB=1.723, [valid] time=0.35 seconds, total_count=0, gpu_max_cached_mem_GB=1.723 [4090c] 2023-11-23 05:36:51,229 (build_trainer:406) INFO: There are no improvements in this epoch [4090c] 2023-11-23 05:36:51,271 (build_trainer:470) INFO: The model files were removed: /home/cnhis/whyme/FunASR/egs_modelscope/asr/TEMPLATE/checkpoint/3epoch.pb [4090c] 2023-11-23 05:36:51,271 (build_trainer:474) WARNING: The gradients at all steps are invalid in this epoch. Something seems wrong. This training was stopped at 4epoch

sunneam avatar Nov 23 '23 05:11 sunneam