flashlight icon indicating copy to clipboard operation
flashlight copied to clipboard
verified publisher icon flashlight

Different log base for scoring calculation with kenlm

Open fann1993814 opened this issue 4 years ago • 0 comments

Question

The log prob score in KenLM is caculated by log_10. (base is 10) https://github.com/kpu/kenlm/issues/191#issuecomment-439011400

However, the decode processes (Building the Trie tree) use log_e (base is e -> natural logarithm) for most case calculations, such as TrieLogAdd in Trie.cpp . https://github.com/flashlight/flashlight/blob/fa118d5f3c4bcde0022855ea887b46d27bd60304/flashlight/lib/text/decoder/Trie.cpp#L65 Is it make incorrect scores?

fann1993814 avatar Oct 03 '21 05:10 fann1993814