Quality of the answers

[pope-llamas0g]

trafficstars

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

azd version?

run azd version and copy paste here.

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

The quality of the answers are not great sometimes. I have uploaded 3 PDFs and when asking about the content, the information is retrieved partially, so it is partially correct.

[pamelafox]

Have you gone through the steps here about debugging answer quality? https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/customization.md#improving-answer-quality

It sounds like you're seeing that it's not doing a good job searching. See https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/customization.md#improving-azure-ai-search-results and let us know if you have follow-up questions.

[pope-llamas0g]

Hi Pamela, thanks for the swift reply. I increased the number of retrieves from 3 to 5, and also decreased the temperature to 0.3 and it improved the quality of the answers a bit. When further asking more questions, I noticed that started going out of the sources and citing non existing sources. I modified the original system prompt a bit.

Which other ways could I debug the q&as?

Also, is there a way to check on the portal or logs? all the questions and answers from all users?

Thanks.

[pamelafox]

Glad those helped! You could check out the evaluation tool: https://github.com/Azure-Samples/ai-rag-chat-evaluator

To find the answers for users, I believe you should be able to use logging.info() and see those logs in Azure Monitor Application Insights, since we have OpenTelemetry configured. I need to verify that myself, another developer asked about that today in a separate issue.

[pope-llamas0g]

Hi Pamela, I am still having serious issues with quality of the answers. Have you experienced any of this? when you try to ask tricky questions, the answers sometimes are being combined with pre-trained data 2002 and with the new data from the uploaded documents. It is also citing non-existing documents [doc1.pdf] [doc2.pdf]

I wonder if anyone else has done rigorous testing to the quality of the answers.

I haven't tried this evaluation tool, I am not sure it can help. Do you have a video to set it up?