chat-with-your-data-solution-accelerator icon indicating copy to clipboard operation
chat-with-your-data-solution-accelerator copied to clipboard

Allow image based questions from `/api/conversation/azure_byod`

Open adamdougal opened this issue 1 year ago • 0 comments

Required by https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/issues/323

Description

When calling the /api/conversation/azure_byod endpoint, generate embeddings of the question using computer vision and query the image embeddings using the question embeddings. We should continue to create question embeddings using text-embeddings-ada-002 in addition to be used to search the image description.

Tasks

  • [ ] Generate question embeddings using computer vision
  • [ ] Query the search index using the generated embeddings
  • [ ] Pass any returned images to gpt-4-vision along with the chat

adamdougal avatar Apr 24 '24 07:04 adamdougal