Generate and store image embeddings in search index

[adamdougal]

Required by https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/issues/323

Description

Generate image embeddings using computer vision and store them in a modified search index.

Tasks

• [x] Investigate what needs to change with the search index and how to encorporate that into CWYDSA
  • [x] POC: https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/786
  • [x] ADR: https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/788
• [ ] Stop using Langchain for creating and populating index
• [ ] Generate image embeddings using computer vision
• [ ] Store image embeddings in search index

[adamdougal]

Update 26th April:

• I'm investigating what changes are going to be required to the app code and index to be able to store and query an additional image embedding
• I've hit a problem where it does not appear that LangChain has support for Azure AI Computer Vision OR has the ability to query AzureSearch for more than one vector embedding.
• One option that might work is to have two separate indexes and combine the search results in the application code. I'm not 100% sure if this is viable as yet
• Another option is to remove the use of LangChain in the QuestionAnswerTool and AzureSearchHelper
  • We would still keep the LangChain orchestration option, but it would only be used in the LangChainAgent
  • This has also been done (for now) in the azure-sample-openai-demo repo
  • Benefits:
    • This would give us full control over how we query and store data in Azure AI Search
    • We wouldn't be blocked on LangChain updates to use the latest features from OpenAI and Azure
    • Potentially simpler code base
  • Downsides:
    • Potentially significant breaking changes required

My next steps are to investigate if removing LangChain allows easier integration of vision as well as seeing what else it might affect. If all goes well, I'll raise an ADR to discuss futher with the team.

[adamdougal]

Update 1st May:

• We are proceeding with removing LangChain from the tools
• Before starting to make this change, I am going to expand the current functional tests to ensure we are testing index creation. This will help ensure no unexpected changes are being made to the index.

[adamdougal]

13th May:

Mini code review:

• [x] Include original exception when catching and throwing
• [x] Move inline patches to annotations
• [x] Make sure new files are pep8 compliant

Get wider feedback:

• [ ] The use of pytest-httpserver in unit tests

[adamdougal]

14th May Update:

• A PR has been raised to add the call to computer vision to generate embeddings of the images
• Next steps are to take these embeddings and store them in a modified index