chat-with-your-data-solution-accelerator icon indicating copy to clipboard operation
chat-with-your-data-solution-accelerator copied to clipboard

Search image data from `/api/conversation/custom` when using `langchain`

Open adamdougal opened this issue 1 year ago • 0 comments

Required by https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/issues/323

Description

When calling the /api/conversation/custom endpoint and using the langchain orchestrator, generate embeddings of the question using computer vision and query the image embeddings using the question embeddings. We should continue to create question embeddings using text-embeddings-ada-002 in addition to be used to search the image description.

Tasks

  • [ ] Generate question embeddings using computer vision
  • [ ] Query the search index using the generated embeddings
  • [ ] Pass any returned images to gpt-4-vision along with the chat

adamdougal avatar Apr 24 '24 07:04 adamdougal