kneser-ney
kneser-ney copied to clipboard
Does not support out of vocabulary words
If this language model is trained on one corpus (e.g. gutenberg) and applied to another (e.g. brown), it is very likely to encounter out of vocabulary words or unseen ngrams. And then this happens:
TypeError: unsupported operand type(s) for +=: 'float' and 'NoneType'