Settings Management

[stewit]

trafficstars

Question

Pydantic can be used for "settings management" (see https://docs.pydantic.dev/latest/concepts/pydantic_settings/) which means parsing/validating from environment variables and .env files.

Are you planning to support such a feature, possibly as an option, since it may require python-dotenv dependency?

Are you interested in a PR for this?

[aedify-swi]

You mean something like this?

import os
import pathlib
import msgspec

from dotenv import dotenv_values

PostgresDsn = Annotated[
    str,
    msgspec.Meta(pattern=r"^postgresql\+psycopg:\/\/[^:]+:[^@]+@[^:]+(:\d+)?\/\w+$"),
]

class SampleSettings(msgspec.Struct, frozen=True):
    _prefix: ClassVar[str] = "sample"

    postgres_dsn: PostgresDsn
    name: str

    @classmethod
    def from_env(
        cls,
        env_files: list[pathlib.Path | str] = [".env"],
    ) -> "SampleSettings":
        env = load_environment(env_files)
        settings_env = {
            k.replace(f"{cls._prefix}_", ""): v
            for k, v in env.items()
            if k.startswith(cls._prefix)
        }
        return msgspec.convert(settings_env, SampleSettings, strict=False)
    

def load_environment(env_files: list[pathlib.Path | str]) -> dict[str, str]:
    env: dict[str, str | None] = {}
    for f in env_files:
        env.update(**dotenv_values(f))
    env.update(**os.environ)
    return {k.lower(): v for k, v in env.items() if v is not None}

if __name__ == "__main__":
    settings = SampleSettings.from_env()

Of course you can just put that into post init and you have the (terrible) auto-magical behaviour of pydantic-settings.

[jack-mcivor]

To extend the above, you can handle more complex types (eg. tuple or nested structs) by making them JSON encoded. This uses the trick mentioned here https://github.com/jcrist/msgspec/issues/375#issuecomment-1520301756

from collections.abc import Iterable
from typing import Annotated, ClassVar, Generic, TypeVar

class JSONStr(Generic[T]):
    """A wrapper type for handling nested JSON values (eg. JSON-in-JSON)
    See https://github.com/jcrist/msgspec/issues/375#issuecomment-1520301756
    """
    value: T

    def __init__(self, value: T):
        self.value = value

    def __repr__(self) -> str:
        return f"JSONStr({self.value})"

def dec_hook(type, value):
    if getattr(type, "__origin__", None) is JSONStr:
        inner_type = type.__args__[0]
        return JSONStr(msgspec.json.decode(value, type=inner_type))
    raise TypeError(f"{type} is not supported")

class User(msgspec.Struct, frozen=True):
    name: str
    age: int

class SampleSettings(msgspec.Struct, frozen=True):
    _prefix: ClassVar[str] = "sample"

    postgres_dsn: PostgresDsn
    name: str
    ids: JSONStr[tuple[int, ...]]
    user: JSONStr[User]

    @classmethod
    def from_env(
        cls,
        env_files: Iterable[pathlib.Path | str] = (".env", ),
    ) -> "SampleSettings":
        env = load_environment(env_files)
        settings_env = {k.replace(f"{cls._prefix}_", ""): v for k, v in env.items() if k.startswith(cls._prefix)}
        return msgspec.convert(settings_env, SampleSettings, strict=False, dec_hook=dec_hook)

def load_environment(env_files: Iterable[pathlib.Path | str]) -> dict[str, str]:
    env: dict[str, str | None] = {}
    for f in env_files:
        env.update(**dotenv_values(f))
    env.update(**os.environ)
    return {k.lower(): v for k, v in env.items() if v is not None}

if __name__ == "__main__":
    settings = SampleSettings.from_env()
    print(settings)
    # Access the nested JSON properties with `.value`
    print(settings.ids, settings.user.value.name)

Example usage

sample_ids='[1,2]' sample_user='{"name":"alice", "age":100}' sample_name=bob sample_postgres_dsn='postgresql+psycopg://username:password@hostname:5432/database' python <script>.py

[stewit]

Hello, thanks you both for your examples. Helped me clarify my wishes:

• Actually I am not interested in the "automagic" functionality that uses prefixed field names as environment variables.
• I prefer to explicitely set the environment variable name per field. It would be nice to have that baked into a subclass of Struct, instead of having to maintain a separate mapping from explicit environment variable names to field names.
• Of course, the parsing from env files (and with higher priority the actual enviroment) as a classmethod of that subclass would be very convenient.

[rafalkrupinski]

IMO this is out of scope, even pydantic delivers it in a separate package. Do you really need objects optimized for memory and speed to keep your settings? Settings are typically read once per process, but if you do, feel free to create a package.

I'd recommend https://pypi.org/project/dataclass-settings/