[Bug]: Azure Open AI returns 500 model error when used in OpenAIAgent

[akabeera]

Bug Description

I am using an Open AI deployment on Azure and when running in an OpenAIAgent chat engine and it's returning HTTP/1.1 500 model_error almost every time.

A few observations:

1. My Agent runs successfully if I hit OpenAI directly
2. If I query my rag using AzureOpenAI it works fine
3. If I hit my azure deployment directly from postman, it run successfully
4. I'm seeing the same issue with both llama_index 0.10.56 and 0.10.55
5. The errors are flaky, my 5% of the calls are successful, but most are failures

Version

0.10.55

Steps to Reproduce

This is my setup:

llm = AzureOpenAI(model=model, deployment=AZURE_DEPLOYMENT_NAME, api_key=API_KEY, azure_endpoint=AZURE_OPENAI_ENDPOINT, api_version=OPENAI_API_VERSION)
    tool_service_context = ServiceContext.from_defaults(
        llm=llm,
        embed_model=embedding_model,
        node_parser=node_parser
    )

.
.
.
llm used in query engines for tools
.
.
.

    top_level_sub_tools = [
        QueryEngineTool(
            query_engine=engine1,
            metadata=ToolMetadata(
                name="engine1",
                description="""some description""".strip())
        ),
        QueryEngineTool(
            query_engine=engine2,
            metadata=ToolMetadata(
                name="engine2",
                description="""\some description here""".strip())
        )
    ]

chat_engine = AzureOpenAI(model=model, deployment=AZURE_DEPLOYMENT_NAME, api_key=API_KEY, azure_endpoint=AZURE_OPENAI_ENDPOINT, api_version=OPENAI_API_VERSION)

    tools = top_level_sub_tools + some_function_call_tools


chat_engine = OpenAIAgent.from_tools(
        tools=tools,
        llm=chat_llm,
        chat_history=chat_history,
        verbose=True,
        system_prompt=SYSTEM_MESSAGE_GENERIC.format(curr_date=curr_date)
    )

Relevant Logs/Tracbacks

2024-07-23 01:42:33,006 - INFO - HTTP Request: POST https://.....openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2023-03-15-preview "HTTP/1.1 500 model_error"
2024-07-23 01:42:33,007 - INFO - Retrying request to /chat/completions in 0.980118 seconds
2024-07-23 01:42:53,728 - INFO - HTTP Request: POST https://......openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2023-03-15-preview "HTTP/1.1 500 model_error"
2024-07-23 01:42:53,729 - INFO - Retrying request to /chat/completions in 1.512375 seconds
2024-07-23 01:43:15,511 - INFO - HTTP Request: POST https://......openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2023-03-15-preview "HTTP/1.1 500 model_error"
2024-07-23 01:43:15,512 - INFO - Retrying request to /chat/completions in 3.923435 seconds
2024-07-23 01:43:34,617 - INFO - HTTP Request: POST https://......openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2023-03-15-preview "HTTP/1.1 500 model_error"
error occurred in /chat endpoint: Error code: 500 - {'error': {'message': 'The server had an error processing your request. Sorry about that! You can retry your request, or contact us through an Azure support request at: https://go.microsoft.com/fwlink/?linkid=2213926 if you keep seeing this error. (Please include the request ID 77c0d6e4-150a-4b70-a132-3dfbec165c8e in your email.)', 'type': 'server_error', 'param': None, 'code': None}}