RemoveKaldiNonWords transformation not working as expected

[tomassykora]

Hello, when trying to use the RemoveKaldiNonWords transformation, I get different results when comparing these text pairs:

• <unk> xx to xx -> 0.5 WER (0.33 when not using SentencesToListOfWords)
• <unk>xx to xx -> 0.0 WER

I'd expect both to be zero when using RemoveKaldiNonWords. Is it an actual bug or am I not understanding the usage correctly? I've tried different order combination of the transformations in the below code, the results were always the same.

transformation = jiwer.Compose([
    jiwer.RemoveKaldiNonWords(),
    jiwer.RemoveMultipleSpaces(),
    jiwer.RemoveWhiteSpace(replace_by_space=True),
    jiwer.SentencesToListOfWords(word_delimiter=' '),
])
wer = jiwer.wer(
    truth,
    hypothesis,
    truth_transform=transformation, 
    hypothesis_transform=transformation,
)

[nikvaessen]

Your transformation should also include jiwer.RemoveEmptyStrings.

import jiwer

truth = "hello"
hypothesis = "hello <unk>"

buggy_custom_transform = jiwer.Compose(
    [
        jiwer.RemoveKaldiNonWords(),
        jiwer.RemoveMultipleSpaces(),
        jiwer.RemoveWhiteSpace(replace_by_space=True),
        jiwer.SentencesToListOfWords(word_delimiter=" "),
    ]
)
working_custom_transform = jiwer.Compose(
    [
        jiwer.RemoveKaldiNonWords(),
        jiwer.RemoveMultipleSpaces(),
        jiwer.RemoveWhiteSpace(replace_by_space=True),
        jiwer.RemoveEmptyStrings(),
        jiwer.SentencesToListOfWords(word_delimiter=" "),
    ]
)

buggy_error_rate = jiwer.wer(
    truth,
    hypothesis,
    truth_transform=buggy_custom_transform,
    hypothesis_transform=buggy_custom_transform,
)
correct_error_rate = jiwer.wer(
    truth,
    hypothesis,
    truth_transform=working_custom_transform,
    hypothesis_transform=working_custom_transform,
)

print(f'after transform: truth={buggy_custom_transform(truth)}, hypothesis:{buggy_custom_transform(hypothesis)}')
print(f'after transform: truth={working_custom_transform(truth)}, hypothesis:{working_custom_transform(hypothesis)}')
print("buggy wer:", buggy_error_rate)
print("correct wer:", correct_error_rate)

after transform: truth=['hello'], hypothesis:['hello', '']
after transform: truth=['hello'], hypothesis:['hello']
buggy wer: 1.0
correct wer: 0.0