linux:Ubuntu 20.04.4
python=3.8.18
torch=2.0.1
funasr=0.8.2
modelscope=1.9.3
训练e-branchformer模型的时候，在130个epoch的时候强制停止训练，然后使用valid.acc.best.pb文件去识别测试集，得到了全是，将测试集换成训练集一样全是，是什么原因，该怎么去解决？
最后一个完整的epoch的训练日志：
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:14:21,657 (build_trainer:248) INFO: 132/180epoch started. Estimated time to finish: 3.555612834232201 hours
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:14:36,118 (build_trainer:730) INFO: 132epoch:train:1-50batch:136814num_updates: iter_time=0.041, forward_time=0.092, loss_ctc=2.658, loss_att=1.954, acc=0.927, loss=2.166, backward_time=0.060, optim_step_time=0.021, optim0_lr0=2.529e-04, train_time=0.289
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:14:47,963 (build_trainer:730) INFO: 132epoch:train:51-100batch:136864num_updates: iter_time=1.629e-04, forward_time=0.091, loss_ctc=3.585, loss_att=2.472, acc=0.912, loss=2.806, backward_time=0.060, optim_step_time=0.023, optim0_lr0=2.529e-04, train_time=0.237
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:14:59,943 (build_trainer:730) INFO: 132epoch:train:101-150batch:136914num_updates: iter_time=2.169e-04, forward_time=0.091, loss_ctc=3.075, loss_att=2.135, acc=0.921, loss=2.417, backward_time=0.063, optim_step_time=0.021, optim0_lr0=2.528e-04, train_time=0.240
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:15:11,665 (build_trainer:730) INFO: 132epoch:train:151-200batch:136964num_updates: iter_time=1.632e-04, forward_time=0.089, loss_ctc=3.078, loss_att=2.216, acc=0.916, loss=2.474, backward_time=0.061, optim_step_time=0.022, optim0_lr0=2.528e-04, train_time=0.234
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:15:23,406 (build_trainer:730) INFO: 132epoch:train:201-250batch:137014num_updates: iter_time=1.592e-04, forward_time=0.092, loss_ctc=3.353, loss_att=2.427, acc=0.911, loss=2.705, backward_time=0.060, optim_step_time=0.020, optim0_lr0=2.527e-04, train_time=0.235
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:15:35,236 (build_trainer:730) INFO: 132epoch:train:251-300batch:137064num_updates: iter_time=1.583e-04, forward_time=0.091, loss_ctc=2.722, loss_att=1.844, acc=0.930, loss=2.107, backward_time=0.061, optim_step_time=0.021, optim0_lr0=2.527e-04, train_time=0.237
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:15:47,053 (build_trainer:730) INFO: 132epoch:train:301-350batch:137114num_updates: iter_time=1.675e-04, forward_time=0.092, loss_ctc=3.003, loss_att=2.071, acc=0.926, loss=2.351, backward_time=0.061, optim_step_time=0.021, optim0_lr0=2.526e-04, train_time=0.236
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:15:58,965 (build_trainer:730) INFO: 132epoch:train:351-400batch:137164num_updates: iter_time=1.657e-04, forward_time=0.093, loss_ctc=2.838, loss_att=2.002, acc=0.925, loss=2.253, backward_time=0.067, optim_step_time=0.021, optim0_lr0=2.526e-04, train_time=0.238
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:16:11,001 (build_trainer:730) INFO: 132epoch:train:401-450batch:137214num_updates: iter_time=1.604e-04, forward_time=0.093, loss_ctc=2.926, loss_att=2.101, acc=0.920, loss=2.349, backward_time=0.064, optim_step_time=0.021, optim0_lr0=2.525e-04, train_time=0.241
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:16:23,171 (build_trainer:730) INFO: 132epoch:train:451-500batch:137264num_updates: iter_time=1.739e-04, forward_time=0.098, loss_ctc=2.999, loss_att=2.125, acc=0.922, loss=2.387, backward_time=0.063, optim_step_time=0.021, optim0_lr0=2.525e-04, train_time=0.243
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:16:35,015 (build_trainer:730) INFO: 132epoch:train:501-550batch:137314num_updates: iter_time=1.905e-04, forward_time=0.091, loss_ctc=2.944, loss_att=2.123, acc=0.920, loss=2.369, backward_time=0.063, optim_step_time=0.021, optim0_lr0=2.525e-04, train_time=0.237
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:16:47,047 (build_trainer:730) INFO: 132epoch:train:551-600batch:137364num_updates: iter_time=1.538e-04, forward_time=0.095, loss_ctc=2.936, loss_att=2.079, acc=0.924, loss=2.336, backward_time=0.062, optim_step_time=0.020, optim0_lr0=2.524e-04, train_time=0.241
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:16:58,982 (build_trainer:730) INFO: 132epoch:train:601-650batch:137414num_updates: iter_time=1.575e-04, forward_time=0.092, loss_ctc=2.832, loss_att=1.994, acc=0.924, loss=2.245, backward_time=0.062, optim_step_time=0.024, optim0_lr0=2.524e-04, train_time=0.239
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:17:10,652 (build_trainer:730) INFO: 132epoch:train:651-700batch:137464num_updates: iter_time=1.461e-04, forward_time=0.091, loss_ctc=3.001, loss_att=2.102, acc=0.926, loss=2.371, backward_time=0.061, optim_step_time=0.020, optim0_lr0=2.523e-04, train_time=0.233
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:17:22,780 (build_trainer:730) INFO: 132epoch:train:701-750batch:137514num_updates: iter_time=1.531e-04, forward_time=0.097, loss_ctc=3.268, loss_att=2.340, acc=0.917, loss=2.619, backward_time=0.063, optim_step_time=0.022, optim0_lr0=2.523e-04, train_time=0.243
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:17:34,765 (build_trainer:730) INFO: 132epoch:train:751-800batch:137564num_updates: iter_time=1.555e-04, forward_time=0.097, loss_ctc=2.990, loss_att=2.126, acc=0.922, loss=2.385, backward_time=0.071, optim_step_time=0.020, optim0_lr0=2.522e-04, train_time=0.240
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:17:46,680 (build_trainer:730) INFO: 132epoch:train:801-850batch:137614num_updates: iter_time=1.405e-04, forward_time=0.092, loss_ctc=2.995, loss_att=2.150, acc=0.918, loss=2.403, backward_time=0.077, optim_step_time=0.019, optim0_lr0=2.522e-04, train_time=0.238
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:17:58,430 (build_trainer:730) INFO: 132epoch:train:851-900batch:137664num_updates: iter_time=1.393e-04, forward_time=0.093, loss_ctc=2.699, loss_att=1.947, acc=0.927, loss=2.173, backward_time=0.066, optim_step_time=0.020, optim0_lr0=2.521e-04, train_time=0.235
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:18:10,062 (build_trainer:730) INFO: 132epoch:train:901-950batch:137714num_updates: iter_time=1.398e-04, forward_time=0.091, loss_ctc=3.145, loss_att=2.234, acc=0.919, loss=2.507, backward_time=0.059, optim_step_time=0.020, optim0_lr0=2.521e-04, train_time=0.233
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:18:21,924 (build_trainer:730) INFO: 132epoch:train:951-1000batch:137764num_updates: iter_time=1.387e-04, forward_time=0.094, loss_ctc=3.283, loss_att=2.276, acc=0.913, loss=2.578, backward_time=0.059, optim_step_time=0.020, optim0_lr0=2.520e-04, train_time=0.237
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:18:46,165 (build_trainer:299) INFO: 132epoch results: [train] iter_time=0.002, forward_time=0.093, loss_ctc=3.028, loss_att=2.143, acc=0.921, loss=2.408, backward_time=0.063, optim_step_time=0.021, optim0_lr0=2.525e-04, train_time=0.240, time=4 minutes and 10.62 seconds, total_count=137808, gpu_max_cached_mem_GB=9.828, [valid] loss_ctc=11.739, cer_ctc=0.838, loss_att=9.440, acc=0.728, cer=0.169, wer=0.748, loss=10.129, time=13.88 seconds, total_count=16632, gpu_max_cached_mem_GB=9.828
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:18:47,406 (build_trainer:403) INFO: There are no improvements in this epoch
[autodl-container-28da11ab52-63ba08d6] 2023-11-28 19:18:47,419 (build_trainer:467) INFO: The model files were removed: /root/autodl-tmp/FunASR/egs/aishell/e_branchformer/result_ex_0.0005_zixing/exp/baseline_train_asr_e_branchformer_zh_char_exp1/131epoch.pb
使用训练集测试的结果：
2023-11-28 20:39:49,478 (beam_search:376) INFO: end detected at 29
2023-11-28 20:39:49,587 (beam_search:400) INFO: -3.91 * 0.6 = -2.35 for decoder
2023-11-28 20:39:49,587 (beam_search:400) INFO: -4.43 * 0.4 = -1.77 for ctc
2023-11-28 20:39:49,587 (beam_search:403) INFO: total log probability: -4.12
2023-11-28 20:39:49,587 (beam_search:404) INFO: normalized log probability: -0.46
2023-11-28 20:39:49,587 (beam_search:405) INFO: total number of ended hypotheses: 77
2023-11-28 20:39:49,588 (beam_search:407) INFO: best hypo:
2023-11-28 20:39:49,592 (asr_inference_launch:215) INFO: uttid: A1_1
2023-11-28 20:39:49,592 (asr_inference_launch:216) INFO: text predictions:
2023-11-28 20:39:49,594 (asr_inference_launch:215) INFO: uttid: A1_1
2023-11-28 20:39:49,594 (asr_inference_launch:216) INFO: text predictions:
2023-11-28 20:39:49,596 (asr_inference_launch:215) INFO: uttid: A1_1
2023-11-28 20:39:49,596 (asr_inference_launch:216) INFO: text predictions:
2023-11-28 20:39:49,598 (asr_inference_launch:215) INFO: uttid: A1_1
2023-11-28 20:39:49,598 (asr_inference_launch:216) INFO: text predictions:
2023-11-28 20:39:49,600 (asr_inference_launch:215) INFO: uttid: A1_1
2023-11-28 20:39:49,600 (asr_inference_launch:216) INFO: text predictions:
使用测试集识别结果：
2023-11-28 19:55:12,355 (beam_search:362) INFO: decoder input length: 83
2023-11-28 19:55:12,355 (beam_search:363) INFO: max output length: 83
2023-11-28 19:55:12,355 (beam_search:364) INFO: min output length: 0
2023-11-28 19:55:18,677 (beam_search:376) INFO: end detected at 24
2023-11-28 19:55:18,788 (beam_search:400) INFO: -3.49 * 0.6 = -2.09 for decoder
2023-11-28 19:55:18,789 (beam_search:400) INFO: -2.89 * 0.4 = -1.16 for ctc
2023-11-28 19:55:18,789 (beam_search:403) INFO: total log probability: -3.25
2023-11-28 19:55:18,789 (beam_search:404) INFO: normalized log probability: -0.46
2023-11-28 19:55:18,789 (beam_search:405) INFO: total number of ended hypotheses: 77
2023-11-28 19:55:18,789 (beam_search:407) INFO: best hypo:
2023-11-28 19:55:18,792 (asr_inference_launch:215) INFO: uttid: A23_331
2023-11-28 19:55:18,793 (asr_inference_launch:216) INFO: text predictions:
2023-11-28 19:55:18,794 (asr_inference_launch:215) INFO: uttid: A23_331
2023-11-28 19:55:18,795 (asr_inference_launch:216) INFO: text predictions:
2023-11-28 19:55:18,796 (asr_inference_launch:215) INFO: uttid: A23_331
2023-11-28 19:55:18,797 (asr_inference_launch:216) INFO: text predictions:
2023-11-28 19:55:18,798 (asr_inference_launch:215) INFO: uttid: A23_331
2023-11-28 19:55:18,799 (asr_inference_launch:216) INFO: text predictions:
2023-11-28 19:55:18,801 (asr_inference_launch:215) INFO: uttid: A23_331
2023-11-28 19:55:18,801 (asr_inference_launch:216) INFO: text predictions:
请问是什么原因
FunASR copied to clipboard
modelscope
