Discrepancies between icefall and sherpa

[ezerhouni]

Following : https://github.com/k2-fsa/icefall/issues/1119

[jingzhaoou]

For the same wav file, I got different transcripts between ./build/binsherpa-onnx in sherpa-onnx and ./build/bin/sherpa-online in sherpa. I found that for the streaming flow, sherpa-onnx explicitly sets snip_edges to false.

https://github.com/k2-fsa/sherpa-onnx/blob/bebc1f1398e77b5b1be00893cd1e0f058924d7bd/sherpa-onnx/csrc/features.cc#L44

However, sherpa simply takes the default snip_edges value, which is true.

https://github.com/csukuangfj/kaldifeat/blob/17861dce6bfe0863f47273283c210248e3b38243/kaldifeat/csrc/feature-window.h#L44

I changed sherpa-onnx to use snip_edges = true. After that, sherpa and sherpa-onnx gave me the same transcripts. My questions are

1. What is the correct values for snip_edges in streaming and offline flows?
2. Should we make things consistent between sherpa and sherpa-onnx?

I appreciate your help!

[jingzhaoou]

Once snip_edges is set to the same, sherpa and sherpa-onnx give the same output for the streaming flow. The transcripts are the same. The joiner outputs are the same as well (except for some 10^-6 rounding errors).

[csukuangfj]

What is the correct values for snip_edges in streaming and offline flows?

Kaldi uses snip_edges=true by default.

In next-gen Kaldi, we are using lhotse to compute features, which hardcodes snip_edges == false.

I think it won't make a big difference about whether to set it to false or true.

---

Should we make things consistent between sherpa and sherpa-onnx?

Yes, I agree that we should make it consistent so that it makes debugging easier. Could you make a PR to fix sherpa-onnx so that it uses snip_edges=false? We use false since the model is trained using features computed with snip_edges=false.

[jingzhaoou]

@csukuangfj thanks a lot for sharing the background. More than happy to contribute. sherpa-onnx already sets snip_edges = false. Do you want me to create a PR and fix sherpa (not sherpa-onnx) to use snip_edges = false?

[jingzhaoou]

"I think it won't make a big difference about whether to set it to false or true.": @csukuangfj for the same model, we sometimes observed absolute WER differences of 1 %to 2% between sherpa and sherpa-onnx, which is most probably due to the different snip_edges settings.

[csukuangfj]

Do you want me to create a PR and fix sherpa (not sherpa-onnx) to use snip_edges = false?

Yes, please.

[jingzhaoou]

@csukuangfj Awesome, I will create a PR on sherpa then. Thanks a lot!