Knover icon indicating copy to clipboard operation
Knover copied to clipboard

Published 20 hours ago • verified publisher icon PaddlePaddle

PLATO-XL Perplexity

Open jasonwu0731 opened this issue 2 years ago • 3 comments

Hello, Can you suggest how to get the perplexity of a target response using your model? What is the score in the prediction about? Thanks!

jasonwu0731 avatar Feb 25 '22 23:02 jasonwu0731

Do you want to calcuate the ppl of a target response, or the ppl of a dataset?

The score in the prediction is the log likelihood of generated response (length average).

sserdoubleh avatar Feb 26 '22 16:02 sserdoubleh

Perplexity of a dataset could be helpful. Thank you!

jasonwu0731 avatar Feb 26 '22 16:02 jasonwu0731

https://github.com/PaddlePaddle/Knover/pull/118 There is a demo about how to evaluate PLATO-XL on a dataset. ./scripts/local/job.sh ./projects/PLATO-XL/eval.conf Because PLATO-XL is too large, you need to run distributed evaluation. And some instances are dropped if the number is less than batch_size * num_gpus.

sserdoubleh avatar Feb 28 '22 15:02 sserdoubleh