[RFE] support LiteLLM Azure OpenAI Entra ID authentication

[thoraxe]

A lot of organizations that use Azure OpenAI are going to want to use Entra ID for authentication. LiteLLM already supports this: https://litellm.vercel.app/docs/providers/azure#entrata-id---use-tenant_id-client_id-client_secret

It doesn't appear that HolmesGPT knows how to support this, looking at: https://github.com/robusta-dev/holmesgpt/blob/master/holmes/core/llm.py

[nherment]

I suspect Holmes already support this as litellm accepts the AZURE_AD_TOKEN en var for the AD token. This is documented here for Azure: https://docs.robusta.dev/master/configuration/ai-analysis.html#azure-ai

I will double check (likely tomorrow) that it is working as expected and will revert back here.

[thoraxe]

There doesn't appear to be a way to pass through the tenant ID or application ID given the way that LiteLLM seems to want that data presented to it. I can give it a test locally.

[thoraxe]

Was suggested to try:

AZURE_API_VERSION=2024-02-15-preview AZURE_API_BASE=https://xxx.azure.com/ AZURE_TENANT_ID=xxx AZURE_CLIENT_ID=xxx AZURE_CLIENT_SECRET=xxx poetry run python3 holmes.py  ask --model="azure/gpt-4o-mini" "can you fix the pod broken-pod in the namespace broken for me?"

This results in:

Exception: model azure/gpt-4o-mini requires the following environment variables: ['AZURE_API_BASE', 'AZURE_API_VERSION', 'AZURE_API_KEY']

[nherment]

Hi @thoraxe , Can you try this branch: https://github.com/robusta-dev/holmesgpt/pull/210/files ?

There are 2 main changes:

• Holmes no longer intercepts and blocks litellm calls even if litellm.validate() call fails. This means you will get the litellm errors bubbled back up.
• Holmes will not set API key or base url if AZURE_API_BASE is detected. It will be up to litellm to read the env vars and use them.

There is an additional change that you can try if the above is still not enough:

export HOLMES_FORCE_AZURE_LITELLM_VARS=1
export AZURE_AD_BEARER_TOKEN_PROVIDER="https://cognitiveservices.azure.com/.default"

This will capture the azure env vars and explicitly pass them to litellm. In addition, this will pass azure_ad_token_provider to litellm as documented here. However I could not make that work but it likely due to the Azure setup I used.

[nherment]

Indeed LiteLLM does not support not having a ad_token. This should be fixed in an upcoming change: https://github.com/BerriAI/litellm/issues/6790. We will revisit support for this once litellm releases the fix.

In the meantime, you could host litellm as a proxy as this part works as expected and then make Holmes use that proxy.