pinecone-io
[Feature] Obtain F score directly from the chat engine
Is this your first time submitting a feature request?
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing functionality
Describe the feature
I am able to get the F score using the knowledge base query object:
query_result = knowledge_base.query([Query(text=query)]) f_score = f"{query_result[0].documents[0].score * 100:.2f}%"
However, I cannot do the same when I use the chat engine directly because the following response object has no score component.
response = chat_engine.chat( messages=[UserMessage(content=query)], )
Would appreciate it if we could get the F score in the response body of the chat engine as well.
Describe alternatives you've considered
The alternative is to go directly to the knowledge base query method, but that misses out on the benefits that the context engine and chat engine provide.
Who will this benefit?
Everyone who wants to understand the F score of their query to benchmark performance and approximate confidence level in the output.
Are you interested in contributing this feature?
Sure, but would appreciate guidance (pair programming maybe?)
Anything else?
No response
hi @NB-123
How do you see the score represented? when you are using knowledge base directly, you basically have the documents the returned from the search, hence you get score for each, with the "chat" you are getting the model response, then what should the score be representing? is it the score of the documents that returned? or the ones the model used (assuming we know exactly which)
what would be ideal request -> response in your mind?
