Builtin method for Azure Cognitive Search?

[tyler-suard-parker]

I noticed there was a module for searching Qdrant, it would be nice to have one too for Azure Cognitive Search.

[sonichi]

It might be as easy as adding the right configuration in the llm_config because I remember seeing a similar question which uses Azure Cognitive Search in an API compatible with openai inference.

[joao-d-oliveira]

I would also be very happy for such solution. I can't find any documentation related with Azure Cognitive Search and AutoGen.

[blakedwbrownie]

Agreed - would be great to have an example of how to use AutoGen with Azure Cognitive/AI Search. Not finding any documentation on it. I think we might be able to create a custom implementation of the RetrieveUserProxyAgent like they did with Qdrant. But given this is a microsoft backed project, I imagine a dedicated Azure AI Search implementation should be on the roadmap

[bsonnek]

I'm surprised this isn't built into Autogen already, is anyone actively working on adding Azure AI Search RAG Agents?

Adding the completions extensions API for DataSources to the llm_config could help to implement this. https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#azure-ai-search

Sample app example for datasources extensions - https://github.com/microsoft/sample-app-aoai-chatGPT/blob/main/app.py

config_list = [ { "model": "gpt-35-turbo-16k", "api_key": os.environ.get("AZURE_OPENAI_API_KEY"), "api_base": os.environ.get("AZURE_OPENAI_ENDPOINT"), "api_type": "azure", "api_version": "2023-05-15", "dataSources": [ { "type": "AzureCognitiveSearch", "parameters": { "endpoint": f"https://{AZURE_SEARCH_SERVICE}.search.windows.net", "key": AZURE_SEARCH_KEY, "indexName": AZURE_SEARCH_INDEX, "fieldsMapping": { "contentFields": parse_multi_columns(AZURE_SEARCH_CONTENT_COLUMNS) if AZURE_SEARCH_CONTENT_COLUMNS else [], "titleField": AZURE_SEARCH_TITLE_COLUMN if AZURE_SEARCH_TITLE_COLUMN else None, "urlField": AZURE_SEARCH_URL_COLUMN if AZURE_SEARCH_URL_COLUMN else None, "filepathField": AZURE_SEARCH_FILENAME_COLUMN if AZURE_SEARCH_FILENAME_COLUMN else None, "vectorFields": parse_multi_columns(AZURE_SEARCH_VECTOR_COLUMNS) if AZURE_SEARCH_VECTOR_COLUMNS else [] }, "inScope": True if AZURE_SEARCH_ENABLE_IN_DOMAIN.lower() == "true" else False, "topNDocuments": int(AZURE_SEARCH_TOP_K), "queryType": query_type, "semanticConfiguration": AZURE_SEARCH_SEMANTIC_SEARCH_CONFIG if AZURE_SEARCH_SEMANTIC_SEARCH_CONFIG else "", "roleInformation": AZURE_OPENAI_SYSTEM_MESSAGE, "filter": filter, "strictness": int(AZURE_SEARCH_STRICTNESS) } } ] }
]

[m-carter1]

I've got it working, but I needed to do some minor changes (I may submit a PR if i have time)

If anyone is interested the basic steps are:

• add the dataSources section to "extra_body" in llm_config
• add "/extensions" to the model name, this is hacky but it means the correct completions url is used

Those 2 steps should create the correct request but the response.choices causes an error, it expects a Message value to check for any function_call's but this is null and instead the message data is in Messages. I created a AzureCognitiveSearchAgent agent and override generate_oai_reply with a fix in there but there may be a better way.

Config Examples

config_list = [ { 'model': "[MODEL_NAME]/extensions", 'api_key': apiKey, 'base_url': base_url, 'api_type': apiType, 'api_version': search_api_version, } ]

llm_config={ "timeout": 600, "cache_seed": 41, "config_list": config_list, "extra_body":{ "dataSources":[ { "type": "AzureCognitiveSearch", "parameters": { "endpoint": "https://[SEARCH_NAME].search.windows.net", "indexName": "[INDEX_NAME]", "key": "[SEARCH_KEY]" } } ] } },

[cforce]

@m-carter1 Please summit MR

[ekzhu]

Is it only for retrieval argumented agent? Would you consider adding the support in the retrieval_utils.py, and creating an AgentCapability so RAG can be added to any agent?

[sonichi]

Is it only for retrieval argumented agent? Would you consider adding the support in the retrieval_utils.py, and creating an AgentCapability so RAG can be added to any agent?

This is a functionality from Azure Cognitive Search. It has an API compatible to Azure OpenAI. Any ConversableAgent can use the llm_config to access Azure Cognitive Search. So I don't think a AgentCapability is necessary here. But documentation, examples are going to be helpful. cc @jackgerrits @m-carter1 @thinkall

[cforce]

Are there any examples meanwhile? @sonichi @m-carter1 @jackgerrits @thinkall

[chengyuliu-msft]

Looking for a solution for Azure AI search, too.

[9hsein5]

Here is suggested changes:

1- Config list needs to follow the following format: config_list = [{ "model": "gpt-35-turbo", "api_key": "***", "base_url": "https://<***>.openai.azure.com/", "api_type": "azure", "api_version": "2024-02-15-preview", "extra_body": { "data_sources": [ { "type": "AzureCognitiveSearch", "parameters": { "endpoint": "https://<***>.search.windows.net", "indexName": "<***>", "key": "<***>", "semanticConfiguration":"<***>", "queryType":"semantic", "fieldsMapping": { "contentFieldsSeparator": "\n", "contentFields": ["<***>"], "filepathField": "<***>", "titleField": "<***>", "urlField": "<***>", "vectorFields": [] }, "filter": null, "inScope": true, "strictness": 3, "topNDocuments": 5 } } ] } } ]

2-Modify message_retrieval in oai Client:

` def message_retrieval( self, response: Union[ChatCompletion, Completion] ) -> Union[List[str], List[ChatCompletionMessage]]: """Retrieve the messages from the response.""" choices = response.choices if isinstance(response, Completion): return [choice.text for choice in choices] # type: ignore [union-attr]

    if TOOL_ENABLED:
        return [  # type: ignore [return-value]
            (
                choice.message  # type: ignore [union-attr] 
                if choice.message.function_call is not None or choice.message.tool_calls is not None  # type: ignore [union-attr]
                else self._format_message_with_citations(choice) #choice.message.content
            )  # type: ignore [union-attr]
            for choice in choices
        ]
    else:
        return [  # type: ignore [return-value]
            choice.message if choice.message.function_call is not None else self._format_message_with_citations(choice)  # type: ignore [union-attr]
            for choice in choices
        ]

`

3- add the following function to oai Client:

`@staticmethod def _format_message_with_citations(choice: List[Dict[str, Any]]) -> str: message_content = choice.message.content try: citations = choice.message.context.get('citations', []) if citations: citation_texts = [] for i, citation in enumerate(citations, start=1): doc_ref = f"[doc{i}]" doc_title = citation.get('title', 'Document Title not provided') doc_path = citation.get('url', 'URL/File path not provided') doc_content = citation.get('content', 'Content not provided') citation_text = f"{i}. [{doc_title}] ({doc_path})\n{doc_content}" citation_texts.append(citation_text)

                # Replace inline citations with doc references
                message_content = message_content.replace(citation['filepath'], doc_ref)
            
            return f"{message_content}\n\nCitations:\n" + "\n\n".join(citation_texts)
    except Exception:
        return choice.message.content`

4-Create a new agent: `import copy from typing import Dict, List, Optional, Tuple, Union

from autogen import OpenAIWrapper from autogen.agentchat import Agent, ConversableAgent

from autogen.code_utils import content_str

from ..._pydantic import model_dump

DEFAULT_LMM_SYS_MSG = """You are a helpful AI assistant.""" DEFAULT_MODEL = "gpt-35-turbo"

class AzureCognitiveSearchAgent(ConversableAgent): DEFAULT_CONFIG = { "model": DEFAULT_MODEL, }

def __init__(
    self,
    name: str,
    system_message: Optional[Union[str, List]] = DEFAULT_LMM_SYS_MSG,
    is_termination_msg: str = None,
    *args,
    **kwargs,
):
    """
    Args:
        name (str): agent name.
        system_message (str): system message for the OpenAIWrapper inference.
            Please override this attribute if you want to reprogram the agent.
        **kwargs (dict): Please refer to other kwargs in
            [ConversableAgent](../conversable_agent#__init__).
    """
    super().__init__(
        name,
        system_message,
        is_termination_msg=is_termination_msg,
        *args,
        **kwargs,
    )
    # call the setter to handle special format.
    self.update_system_message(system_message)
    self._is_termination_msg = (
        is_termination_msg
        if is_termination_msg is not None
        else (lambda x: content_str(x.get("content")) == "TERMINATE")
    )

    # Override the `generate_oai_reply`
    self.replace_reply_func(ConversableAgent.generate_oai_reply, AzureCognitiveSearchAgent.generate_oai_reply)
    self.replace_reply_func(
        ConversableAgent.a_generate_oai_reply,
        AzureCognitiveSearchAgent.a_generate_oai_reply,
    )

def update_system_message(self, system_message: Union[Dict, List, str]):
    """Update the system message.

    Args:
        system_message (str): system message for the OpenAIWrapper inference.
    """
    self._oai_system_message[0]["content"] = self._message_to_dict(system_message)["content"]
    self._oai_system_message[0]["role"] = "system"

def generate_oai_reply(
    self,
    messages: Optional[List[Dict]] = None,
    sender: Optional[Agent] = None,
    config: Optional[OpenAIWrapper] = None,
) -> Tuple[bool, Union[str, Dict, None]]:
    """Generate a reply using autogen.oai."""
    client = self.client if config is None else config
    if client is None:
        return False, None
    if messages is None:
        messages = self._oai_messages[sender]

    # TODO: #1143 handle token limit exceeded error
    response = client.create(context=messages[-1].pop("context", None), messages=messages)

    # TODO: line 301, line 271 is converting messages to dict. Can be removed after ChatCompletionMessage_to_dict is merged.
    extracted_response = client.extract_text_or_completion_object(response)[0]
    
    if not isinstance(extracted_response, str):
        try:
            extracted_response = model_dump(extracted_response)
        except Exception as e:
            extracted_response = "The requested information is not available in the retrieved data. Please try another query or topic."

    return True, extracted_response

`

---

create an AzureCognitiveSearchAgent instance named "retrieval_agent"

retrieval_agent = AzureCognitiveSearchAgent( name="retrieval_agent", llm_config=gpt35_rag_llm_config, )

[tyler-suard-parker]

@9hsein5 do you want to work together on a PR for adding your code to Autogen?

[9hsein5]

@9hsein5 do you want to work together on a PR for adding your code to Autogen?

Yes let's go with it

[tyler-suard-parker]

@sonichi Do you have any recommendations for how we can implement this according to @9hsein5 's code? Do you want us to create a protocol/interface etc? What file or folder would you recommend we add this to?