chat-with-your-data-solution-accelerator icon indicating copy to clipboard operation
chat-with-your-data-solution-accelerator copied to clipboard

Search image data from `/api/conversation/custom`

Open adamdougal opened this issue 1 year ago • 0 comments

Required by https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/issues/323

Description

When calling the /api/conversation/custom endpoint, generate embeddings of the question using computer vision and query the image embeddings using the question embeddings. We should continue to create question embeddings using text-embeddings-ada-002 in addition to be used to search the image description.

Tasks

  • [x] Stop using langchain in the quesiton answer tool and post prompt tool
  • [x] Generate question embeddings using computer vision
  • [x] Query the search index using the generated embeddings
  • [x] Pass any returned images to gpt-4-vision along with the chat
  • [ ] Add a limit on the number of images passed to vision
  • [ ] Update post answering prompt to include images

adamdougal avatar Apr 24 '24 07:04 adamdougal