chat-with-your-data-solution-accelerator
chat-with-your-data-solution-accelerator copied to clipboard
Search image data from `/api/conversation/custom`
Required by https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/issues/323
Description
When calling the /api/conversation/custom endpoint, generate embeddings of the question using computer vision and query the image embeddings using the question embeddings. We should continue to create question embeddings using text-embeddings-ada-002 in addition to be used to search the image description.
Tasks
- [x] Stop using langchain in the quesiton answer tool and post prompt tool
- [x] Generate question embeddings using computer vision
- [x] Query the search index using the generated embeddings
- [x] Pass any returned images to
gpt-4-visionalong with the chat - [ ] Add a limit on the number of images passed to vision
- [ ] Update post answering prompt to include images