kneser-ney icon indicating copy to clipboard operation
kneser-ney copied to clipboard

Does not support out of vocabulary words

Open olekscode opened this issue 6 years ago • 0 comments

If this language model is trained on one corpus (e.g. gutenberg) and applied to another (e.g. brown), it is very likely to encounter out of vocabulary words or unseen ngrams. And then this happens:

TypeError: unsupported operand type(s) for +=: 'float' and 'NoneType'

olekscode avatar Jan 09 '19 00:01 olekscode