In wenetspeech recipe, fast_beam_search_LG almost always get worse WER result than greedy search!

[zhangzhengyireal]

Collecting environment information... k2 version: 1.24.3 Build type: Release Git SHA1: 42e92fdd4097adcfe9937b4d2df7736d227b8e85 Git date: Wed Jun 28 09:50:36 2023 Cuda used to build k2: 11.6 cuDNN used to build k2: 8.2.0 Python version used to build k2: 3.9 OS used to build k2: Ubuntu 20.04.6 LTS CMake version: 3.26.4 GCC version: 7.5.0 PyTorch version used to build k2: 1.13.1+cu116 PyTorch is using Cuda: 11.6 NVTX enabled: True With CUDA: True Disable debug: True Sync kernels : False Disable checks: False Max cpu memory allocate: 214748364800 bytes (or 200.0 GB) k2 abort: False

Resource: https://huggingface.co/pkufool/icefall-asr-zipformer-streaming-wenetspeech-20230615 Testset: wenetspeech/ DEV

Bash command: exp_dir=download/huggingface/icefall-asr-zipformer-streaming-wenetspeech-20230615/exp lang_dir=download/huggingface/icefall-asr-zipformer-streaming-wenetspeech-20230615/data/lang_char decode_method=greedy_search #decode_method=fast_beam_search_LG ./zipformer/decode.py
--epoch ${ep}
--avg ${avg}
--exp-dir ${exp_dir}/
--lang-dir ${lang_dir}
--max-duration 800
--decoding-method ${decode_method}
--blank-penalty ${blank_penalty}
--ngram-lm-scale ${nls}
--ilme-scale ${ilme_scale}
--manifest-dir data/fbank/
--causal 1
--chunk-size ${chunk_size}
--left-context-frames ${left_context}

Result:

In both chunk=16 and chunk=32, I can't get better WER by fast_beam_search_LG.

[danpovey]

Have you tried the LODR method? Also, assuming your LG is based on Chinese words, what is the vocabulary coverage of your dev set like?

[desh2608]

In my experiments, I have always found the "nbest" variations to be better than the one best versions, e.g., fast_beam_search_nbest_LG better than fast_beam_search_LG.

Usually, you would also need to play around with the --beam parameter to balance out insertions vs. deletions. It looks like you have significantly higher deletions at the moment, maybe you can try increasing the beam.