Language Model Score (Perplexity score) of a sentence

[mdasadul]

Hi Thansk for this nice work. @honnibal I am interested to use any of the pre-trained language model to calculate Perplexity score of a sentence. Is there any way to achieve that using this repo?

Thanks

[honnibal]

We don't have that functionality yet unfortunately. I hope we can provide it in a future release.

[zephyrzilla]

Any updates on this?

[tagucci]

It's possible to compute perplexity by GPT-2, whereas Masked Language Models (MLM) such as BERT cannot compute it. In case of MLM, we should consider other metrics. I think pseudo-perplexities is one of the solution. https://arxiv.org/abs/1910.14659