Microsoft KeyVault fetch support

[dvir-ms]

Initial Checks

• [X] I have searched Google & GitHub for similar requests and couldn't find anything
• [X] I have read and followed the docs and still think this feature is missing

Description

Hi

Currently, we use the Pydantic with a wrapper that checks if the field exists (in addition to the env var) in the KeyVault.

This is not ideal as it is done in a separate place.

A new feature of fetching data from KeyVault will be very helpful.

Thanks!

Affected Components

• [ ] Compatibility between releases
• [X] Data validation/parsing
• [ ] Data serialization - .model_dump() and .model_dump_json()
• [ ] JSON Schema
• [ ] Dataclasses
• [ ] Model Config
• [ ] Field Types - adding or changing a particular data type
• [ ] Function validation decorator
• [ ] Generic Models
• [ ] Other Model behaviour - model_construct(), pickling, private attributes, ORM mode
• [ ] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

Selected Assignee: @Kludex

Selected Assignee: @samuelcolvin

[dmontagu]

I'm not familiar with Microsoft KeyVault, but if it is what I would assume it is from the name, this seems like it might be a good candidate for a new SettingsSource in pydantic-settings. This recently-opened PR seems closely related https://github.com/pydantic/pydantic-settings/pull/140, and might serve as a good reference if you wanted to open a PR on pydantic-settings to add support.

[lmmx]

Ah cool, I'm working on introducing keyring which supports Windows Credential Locker. Microsoft Azure KeyVault is a cloud service more like AWS SSM, the keyring library is local (its repo has no hits for the term 'KeyVault' so I don't think it'd be possible to access the secrets through there). My PR is still work in progress! :slightly_smiling_face:

[AndreuCodina]

Can we make progress on this @samuelcolvin? I can lend a hand.

To use Azure Key Vault from localhost, your Microsoft account needs a role, for example, Key Vault Administrator, and then log in to Azure with az login. In an App Service, you only assign the role to it.

This is the code I use to read from Azure Key Vault:

application_settings.py

class ApplicationSettings(BaseSettings):
    model_config = SettingsConfigDict(
        env_nested_delimiter="__",
        extra="ignore"
    )

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        return (
            env_settings,
            dotenv_settings,
            AzureKeyVaultSettingsSource(settings_cls, os.environ["KEY_VAULT__URL"]),
        )

    SQL_SERVER__PASSWORD: str

azure_key_vault_settings_source.py

from typing import Any, Optional

from azure.core.exceptions import ResourceNotFoundError
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
from pydantic.fields import FieldInfo
from pydantic_settings import (
    BaseSettings,
    PydanticBaseSettingsSource,
)


class AzureKeyVaultSettingsSource(PydanticBaseSettingsSource):
    _credential: DefaultAzureCredential
    _secret_client: SecretClient

    def __init__(self, settings_cls: type[BaseSettings], url: str) -> None:
        self._credential = DefaultAzureCredential()
        self._secret_client = SecretClient(vault_url=url, credential=self._credential)
        super().__init__(settings_cls)

    def get_field_value(
        self, field: FieldInfo, field_name: str
    ) -> tuple[Any, str, bool]:
        field_value: Optional[Any] = None

        # It's not possible to use underscores in Azure Key Vault
        secret_name = field_name.replace("_", "-")
        
        try:
            secret = self._secret_client.get_secret(secret_name)  # type: ignore
            field_value = secret.value
        except ResourceNotFoundError:
            field_value = None

        return field_value, field_name, False

    def prepare_field_value(
        self, field_name: str, field: FieldInfo, value: Any, value_is_complex: bool
    ) -> Any:
        return value

    def __call__(self) -> dict[str, Any]:
        data: dict[str, Any] = {}

        for field_name, field in self.settings_cls.model_fields.items():
            field_value, field_key, value_is_complex = self.get_field_value(
                field, field_name
            )
            field_value = self.prepare_field_value(
                field_name, field, field_value, value_is_complex
            )

            if field_value is not None:
                data[field_key] = field_value

        return data

Python packages: azure-keyvault-secrets and azure-identity.

[hramezani]

Thanks @AndreuCodina for the settings source code. You can make a PR for this if you would like. Please consider to add proper test and documentation for this