Issue: Add Azure AI Foundry support for Claude and Grok models

[IanSteno]

Feature Type

I cannot use LiveKit without it

Feature Description

Type: Feature Request

Description

Add support for Claude (Anthropic) and Grok (xAI) models available through Azure AI Foundry's Model Catalog with serverless API deployments.

Azure AI Foundry is Microsoft's platform that provides access to multiple AI providers through a unified, OpenAI-compatible API. This includes:

• Anthropic Claude models: Claude Opus 4.1, Claude Sonnet 4.5, Claude Haiku 4.5
• xAI Grok models: Grok 3, Grok 3 Mini

Motivation

Microsoft Azure now offers these models through Azure AI Foundry, allowing users to:

• Use Claude and Grok models with Azure infrastructure and billing
• Leverage Azure's enterprise features (security, compliance, regional deployment)
• Access multiple AI providers through a single Azure account
• Benefit from serverless deployment with pay-as-you-go pricing

Currently, LiveKit Agents supports:

• Direct OpenAI and Azure OpenAI (through with_azure())
• Direct xAI Grok (through with_xai())
• Direct Anthropic Claude (through separate livekit-plugins-anthropic)

However, there's no way to use these models when they're deployed through Azure AI Foundry.

Proposed Solution

Add a new LLM.with_azure_foundry() static method to the OpenAI plugin that:

1. Supports Azure AI Foundry's serverless API endpoint format
2. Uses OpenAI-compatible API structure (works with OpenAI SDK)
3. Handles Azure-specific authentication (API key or Azure AD tokens)
4. Provides typed model definitions for Claude and Grok models

This follows the existing pattern of multi-provider support in the OpenAI plugin (Cerebras, Perplexity, Together, etc.).

Implementation Details

Model Type Definitions

AzureFoundryClaudeModels = Literal[
    "claude-opus-4-1",
    "claude-sonnet-4-5",
    "claude-haiku-4-5",
    "claude-3-7-sonnet",
    "claude-3-5-haiku",
]

AzureFoundryGrokModels = Literal[
    "grok-3",
    "grok-3-mini",
]

API Method

@staticmethod
def with_azure_foundry(
    *,
    model: str | AzureFoundryModels,
    azure_endpoint: str | None = None,
    api_key: str | None = None,
    azure_ad_token: str | None = None,
    # ... other LLM parameters
) -> LLM

Usage Example

from livekit.plugins import openai
import os

# Using Claude Sonnet via Azure AI Foundry
llm = openai.LLM.with_azure_foundry(
    model="claude-sonnet-4-5",
    azure_endpoint=os.environ["AZURE_FOUNDRY_ENDPOINT"],
    api_key=os.environ["AZURE_FOUNDRY_API_KEY"],
)

# Using Grok via Azure AI Foundry
llm = openai.LLM.with_azure_foundry(
    model="grok-3",
    azure_endpoint=os.environ["AZURE_FOUNDRY_ENDPOINT"],
    api_key=os.environ["AZURE_FOUNDRY_API_KEY"],
)

Why in the OpenAI Plugin?

The OpenAI plugin is already designed as a multi-provider plugin that supports any OpenAI-compatible API:

• ✅ Cerebras, Perplexity, Together, DeepSeek, OctoAI, etc.
• ✅ Azure AI Foundry uses OpenAI-compatible API format
• ✅ Works with the OpenAI Python SDK
• ✅ Keeps all LLM functionality in one place
• ✅ Follows existing patterns in the codebase

Benefits

1. Azure Integration: Enables enterprise users already on Azure to use Claude/Grok
2. Unified Access: Access multiple providers through single Azure subscription
3. Consistent API: Same interface as other providers in OpenAI plugin
4. Type Safety: Typed model definitions prevent typos
5. Environment Variables: Follows existing patterns for configuration

References

• Azure AI Foundry Models Documentation
• Azure AI Model Inference API
• Microsoft Azure Blog: Anthropic's Claude models in Azure
• GeekWire: Microsoft expands AI roster with Anthropic and xAI

Implementation Status

I have already implemented this feature and can submit a PR. The implementation includes:

• Model type definitions (AzureFoundryModels, AzureFoundryClaudeModels, AzureFoundryGrokModels)
• with_azure_foundry() static method with comprehensive documentation
• Environment variable support (AZURE_FOUNDRY_ENDPOINT, AZURE_FOUNDRY_API_KEY)
• OpenAI SDK compatibility using AsyncClient
• All code quality checks pass (ruff format, ruff check)

Branch: feat/azure-foundry-support at https://github.com/aisteno/agents

Testing

The implementation can be tested with:

1. Deployed Claude or Grok model in Azure AI Foundry
2. Serverless API endpoint and key
3. Standard LLM chat operations

I'm available to provide testing guidance and work with the team on any adjustments needed.

Workarounds / Alternatives

No response

Additional Context

No response