infiniflow
[Feature Request]: Add Perplexity score and sufficient_context_score to answers
Is there an existing issue for the same feature request?
- [X] I have checked the existing issues.
Is your feature request related to a problem?
Nope, not a problem.
Describe the feature you'd like
Following idea: Adding the possibility to show evaluations of a models answer (especially the Perplexity score and the has_sufficient _context score) at the end of models answer.
Describe implementation you've considered
In order to implement the perplexity score (at least in case of using azure-open-ai and openai), it's possible to also return the logprobs. Based on the logprobs one is then able to calculate the avrg. perplexity score and show it either below the final response or somewhere else. In order to calculate the has_sufficient_context score, one would only need the model to first answer whether it has sufficient context, but it could only answer with yes or no. Both words will have a specific logprob, which we could use to calculate the prob. that it has enough content.
An example img of the final result (using an own RAG implementation) is provided below (in Additional information)
Documentation, adoption, use case
Especially when playing around with different model parameters/search parameters but also system_prompt, it might be very useful to also see the perplexity score/ retrieval confidence score whenever one wants to. (useful for on-the-fly evaluation)
