chat-with-your-data-solution-accelerator
chat-with-your-data-solution-accelerator copied to clipboard
Allow image based questions from `/api/conversation/azure_byod`
Required by https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/issues/323
Description
When calling the /api/conversation/azure_byod endpoint, generate embeddings of the question using computer vision and query the image embeddings using the question embeddings. We should continue to create question embeddings using text-embeddings-ada-002 in addition to be used to search the image description.
Tasks
- [ ] Generate question embeddings using computer vision
- [ ] Query the search index using the generated embeddings
- [ ] Pass any returned images to
gpt-4-visionalong with the chat