chat-with-your-data-solution-accelerator
chat-with-your-data-solution-accelerator copied to clipboard
Search image data from `/api/conversation/custom` when using `langchain`
Required by https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/issues/323
Description
When calling the /api/conversation/custom endpoint and using the langchain orchestrator, generate embeddings of the question using computer vision and query the image embeddings using the question embeddings. We should continue to create question embeddings using text-embeddings-ada-002 in addition to be used to search the image description.
Tasks
- [ ] Generate question embeddings using computer vision
- [ ] Query the search index using the generated embeddings
- [ ] Pass any returned images to
gpt-4-visionalong with the chat