Azure-Samples
Improve the prompts with Azure OpenAI On your data
Motivation
To ensure the prompts used in this project have parity to the ones used by "on your data" feature so the customers have standard responses if they use this same index from "on your data"
Attached file is the list of prompts that were provided by their Engineering team which are used by “on your data”.
How would you feel if this feature request was implemented?
Requirements
- Update prompts
Tasks
To be filled in by the engineer picking up the issue
- [x] #643
- [x] #648
- [ ] #649
@gmndrg Can you please add more details on what improvements you are looking in current Prompts . Also, will be good to understand your vision for prompt flow evaluation . We do have similar issue #406 , will be good to learn how this issue is diffrent or similar to the other one.
We need to discuss with @ross-p-smith to understand what the relationship is of the "on your data" prompts and prompt flow he adds as part of the feature name. The work we added in the roadmap is to add parity of prompts to the ones used by "on your data" feature so the customers have standard responses if they use this same index from "on your data". @ross-p-smith if you can clarify the PromptFlow part would be great. Thanks. I will send the prompts to @reechamishra
That makes sense @gmndrg, I copied the title from a deck. We have another story (#406) which is about PromptFlow. I've now edited the title.
In summary, this story is just aligning the default prompt between OnYourData and this repo so that people have a standard response from the same index.
@ross-p-smith I have created a branch to try to use the main_prompt that is attached. The code can be seen here.
I have been experimenting with it in the application, and have found the results to be temperamental and unreliable.
- It often says that the information cannot be found in the retrieved data, even though it should be available.
- It often gives citations at the end of the entire paragraph, rather than at the end of each sentence.
- I have got it to answer questions about the prompt itself, e.g. by asking "What is DTE" (I was only able to get this once)
I think the change of the prompt should be looked at in greater detail
It would also be good to have charity over where the two prompts are meant to be used.
I understand that the main prompt is used to answer the user's query after fetching the documents, however I am not sure about the intent prompt. Is this essentially a new tool for the LLM?
I have created a second version using the main prompt, but this time formatting the various system/user/assistant messages correctly rather than using one long user message: https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/commit/9012a1d0a30b14658e5aacaaefe7fe65410d62a0
This seems to produce better results, although it would require a change to the config file
The main prompt has been implemented into the application, and we have guidance on how to use the intent prompt:
call the model to get a list of search intents for the user question (intent_prompt) call the search index and retrieve documents for each search intent, join & rerank the results call the model to get the final answer to the question (main_prompt - includes retrieved chunks)
We may need to re-work how the application works at present, as currently we allow the LLM to determine which function to call, rather than a pre-determined route.
Closing this epic and just leaving the single task if someone wants to pick it up