Add Azure Key Vault settings source

[AndreuCodina]

Add Azure Key Vault settings source.

Doubt: Where should I add the required Python packages (azure-keyvault-secrets==4.8.0 and azure-identity==1.16.0) to use AzureKeyVaultSettingsSource?

This is the infrastructure:

resource "azurerm_resource_group" "pydantic" {
  name     = "pydantic"
  location = "northeurope"
}

resource "azurerm_key_vault" "pydantic" {
  name                          = "kv-pydantic"
  resource_group_name           = "pydantic"
  location                      = "northeurope"
  tenant_id                     = data.azuread_client_config.current.tenant_id
  sku_name                      = "standard"
  soft_delete_retention_days    = 7
  purge_protection_enabled      = false
  enable_rbac_authorization     = true
  public_network_access_enabled = true
}

resource "azurerm_key_vault_secret" "sql_server_password" {
  name         = "SqlServer--Password"
  value        = "SqlServerPassword"
  key_vault_id = azurerm_key_vault.pydantic.id
}

resource "azurerm_key_vault_secret" "my_password" {
  name         = "MyPassword"
  value        = "MyPassword"
  key_vault_id = azurerm_key_vault.pydantic.id
}

Closes #143

[hramezani]

Thanks @AndreuCodina for this PR.

Doubt: Where should I add the required Python packages (azure-keyvault-secrets==4.8.0 and azure-identity==1.16.0) to use AzureKeyVaultSettingsSource?

It has to be added to optional dependencies section

Also, I can't see any test added here. You need to add some tests

[AndreuCodina]

Thanks @AndreuCodina for this PR.

Doubt: Where should I add the required Python packages (azure-keyvault-secrets==4.8.0 and azure-identity==1.16.0) to use AzureKeyVaultSettingsSource?

It has to be added to optional dependencies section

Also, I can't see any test added here. You need to add some tests

How do I make the imports in code? I've created the function import_azure_key_vault, but I'm getting mypy pydantic_settings pydantic_settings/sources.py:14: error: Cannot find implementation or library stub for module named "azure.core.credentials" [import-not-found]

[hramezani]

You need to update the requirements by running make refresh-lockfiles command

[AndreuCodina]

You need to update the requirements by running make refresh-lockfiles command

$ make refresh-lockfiles
`make: *** No rule to make target 'refresh-lockfiles'.  Stop.`

[hramezani]

Please refresh your fork. this command was added recently

[AndreuCodina]

@hramezani is everything ok?

[hramezani]

@AndreuCodina thanks for the update.

I will review it later. My plan is to include it in the next minor release 2.4 if it is ready.

[hramezani]

@AndreuCodina I checked the PR. we have some missed functionality like nested model support, case insensitive support. let me talk with the team and get back to you.

[AndreuCodina]

@AndreuCodina I checked the PR. we have some missed functionality like nested model support, case insensitive support. let me talk with the team and get back to you.

What do I have to do then? I've checked EnvSettingsSource and if I have to copy (and maybe modify some line), this will not be maintainable. I've already duplicated complex code I don't understand.

I can get all available secrets from Azure Key Vault (names and values) and replace -- (these are the symbols used for nested/hierarchy data) with __, but if you require me to add a lot of code just to get secrets from Key Vault, I'd need to use shared code provided by the library.

After this source, I plan to add another source for Azure App Configuration, but I need more guidance here to re-use all code I can.

In summary, in Key Vault I receive values such as "NestedModel--Variable" and I convert them to "NestedModel__Variable", and in App Configuration I directly receive "NestedModel__Variable".

References:

https://learn.microsoft.com/en-us/aspnet/core/security/key-vault-configuration?view=aspnetcore-8.0

Azure Key Vault secret names are limited to alphanumeric characters and dashes. Hierarchical values (configuration sections) use -- (two dashes) as a delimiter, as colons aren't allowed in Key Vault secret names. Colons delimit a section from a subkey in ASP.NET Core configuration. The two-dash sequence is replaced with a colon when the secrets are loaded into the app's configuration.

ASP.NET Core uses ":" to separate configurations/secrets. Using Key Vault, "--" is replaced with ":". Using App Configuration, "__" is replaced with ":".

[hramezani]

I can get all available secrets from Azure Key Vault (names and values) and replace -- (these are the symbols used for nested/hierarchy data) with __, but if you require me to add a lot of code just to get secrets from Key Vault, I'd need to use shared code provided by the library.

No, we don't want to add the code from the library.

Right now, the AzureKeyVaultSettingsSource is doing a very simple thing. looping over model fields and fetching the values. but in other sources like EnvSettingsSource we can load nested model values. for example, by the following model:

    class SubSubModel(BaseSettings):
        dvals: Dict

    class SubModel(BaseSettings):
        vals: List[str]
        sub_sub_model: SubSubModel

    class Cfg(BaseSettings):
        sub_model: SubModel

        model_config = SettingsConfigDict(env_prefix='cfg_', env_nested_delimiter='__')

You can set the envs like cfg_sub_model__vals='["one", "two"]' and cfg_sub_model__sub_sub_model__dvals'='{"three": 4}' and pydantic-settings load the values for the model.

I think we don't have the above functionality in AzureKeyVaultSettingsSource. I also believe we don't have env_prefix support there.

The easy way to have all the functionality is to load all the possible key/value and inherit from EnvSettingsSource. like DotEnvSettingsSource. but it seems there is no easy way to load all the key/values.

So, I would suggest implementing the current functionality that we have in other sources as much as possible and documenting the functionalities that you don't support in AzureKeyVaultSettingsSource.

[AndreuCodina]

I can get all available secrets from Azure Key Vault (names and values) and replace -- (these are the symbols used for nested/hierarchy data) with __, but if you require me to add a lot of code just to get secrets from Key Vault, I'd need to use shared code provided by the library.

No, we don't want to add the code from the library.

Right now, the AzureKeyVaultSettingsSource is doing a very simple thing. looping over model fields and fetching the values. but in other sources like EnvSettingsSource we can load nested model values. for example, by the following model:

    class SubSubModel(BaseSettings):
        dvals: Dict

    class SubModel(BaseSettings):
        vals: List[str]
        sub_sub_model: SubSubModel

    class Cfg(BaseSettings):
        sub_model: SubModel

        model_config = SettingsConfigDict(env_prefix='cfg_', env_nested_delimiter='__')

You can set the envs like cfg_sub_model__vals='["one", "two"]' and cfg_sub_model__sub_sub_model__dvals'='{"three": 4}' and pydantic-settings load the values for the model.

I think we don't have the above functionality in AzureKeyVaultSettingsSource. I also believe we don't have env_prefix support there.

The easy way to have all the functionality is to load all the possible key/value and inherit from EnvSettingsSource. like DotEnvSettingsSource. but it seems there is no easy way to load all the key/values.

So, I would suggest implementing the current functionality that we have in other sources as much as possible and documenting the functionalities that you don't support in AzureKeyVaultSettingsSource.

Ok, I think everything is done. Right now it has the same behavior as .NET.

Edit: env_prefix works, but maybe with a bug. If I have:

class AzureKeyVaultSettings(BaseSettings):
    model_config = SettingsConfigDict(extra='ignore', env_prefix='2-')

    AnotherSecret: str
    another_secret: str = Field(..., alias='AnotherSecret')

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: Any,
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> Tuple[PydanticBaseSettingsSource]:
        return (
            AzureKeyVaultSettingsSource(
                settings_cls,
                os.environ['AZURE_KEY_VAULT_URL'],
                DefaultAzureCredential()
            ),
        )

Then the value of AnotherSecret is got, but the value of another_secret is not because this line is executed and the prefix is not added to the alias:

field_info.append((v_alias, self._apply_case_sensitive(v_alias), False))

Is this anything to change or is it the desired behavior? If it's the desired behavior, I'll remove the prefix support from the Key Vault settings source.

[hramezani]

@AndreuCodina I changed the example in the doc and made it simpler.

Please add AzureKeyVaultSettingsSource to the init and then change the doc example to import AzureKeyVaultSettingsSource from pydantic_settings instead of pydantic_settings.sources

[AndreuCodina]

@AndreuCodina I changed the example in the doc and made it simpler.

Please add AzureKeyVaultSettingsSource to the init and then change the doc example to import AzureKeyVaultSettingsSource from pydantic_settings instead of pydantic_settings.sources

Done. I've removed "```python" in the markdown because if not, the tests fail.

[hramezani]

@AndreuCodina I changed the example in the doc and made it simpler. Please add AzureKeyVaultSettingsSource to the init and then change the doc example to import AzureKeyVaultSettingsSource from pydantic_settings instead of pydantic_settings.sources

Done. I've removed "```python" in the markdown because if not, the tests fail.

it fails because of imports in the code. I've fixed it and returned the "```python"

[hramezani]

Thanks @AndreuCodina