Problematic work with Configuration provider and it's inconsistent behavior and interface

[alexted]

trafficstars

I encountered a strange behaviour of Configuration depending on which source the Configuration object took data from during its initialisation. This leads to a radically different interface of this class for each case. This is unintuitive and confusing and the documentation does not cover this aspect fully.

standard behaviour (loading config from dict):

import boto3
from dependency_injector import containers, providers


class Container(containers.DeclarativeContainer):

    config = providers.Configuration()

    s3_client_factory = providers.Factory(
        boto3.client,
        "s3",
        aws_access_key_id=config.aws.access_key_id,           
        aws_secret_access_key=config.aws.secret_access_key,
        # Notice how here we get direct access to the configuration field data directly
        # after initialising the config object. No additional action is needed!
    )

behaviour specific to loading configurations from pydantic settings object:

import boto3
from dependency_injector import containers, providers


class Container(containers.DeclarativeContainer):

    config = providers.Configuration()
    # But we can't do that here! Because our config object does not contain fields with configuration data 
    # To fill the fields of our object we will have to make one more call:
    config.load()
    # which now isn't working and throws an error, lol: 
    # https://github.com/ets-labs/python-dependency-injector/issues/726

    # The second option is to make another call that will return a pydantic settings object:
    config, = config.get_pydantic_settings()  

    s3_client_factory = providers.Factory(
        boto3.client,
        "s3",
        aws_access_key_id=config.aws.access_key_id,                   
        aws_secret_access_key=config.aws.secret_access_key, 
    )

behaviour specific to loading configurations from INI file:

import boto3
from dependency_injector import containers, providers


class Container(containers.DeclarativeContainer):

    config = providers.Configuration()
    # But we can't do that here! Because our config object does not contain fields with configuration data 
    # To fill the fields of our object we will have to make one more call:
    config.load()
    # which now isn't working and throws an error, lol: 
    # https://github.com/ets-labs/python-dependency-injector/issues/726

    # The second option is to make another call that will return a raw config object:
    config, = config.get_ini_files()

    s3_client_factory = providers.Factory(
        boto3.client,
        "s3",
        aws_access_key_id=config.aws.access_key_id,
        aws_secret_access_key=config.aws.secret_access_key,
    )

behaviour specific to loading configurations from YAML file:

import boto3
from dependency_injector import containers, providers


class Container(containers.DeclarativeContainer):

    config = providers.Configuration()
    # But we can't do that here! Because our config object does not contain fields with configuration data 
    # To fill the fields of our object we will have to make one more call:
    config.load()
    # which now isn't working and throws an error, lol: 
    # https://github.com/ets-labs/python-dependency-injector/issues/726

    # The second option is to make another call that will return a raw config object:
    config, = config.get_yaml_files() 

    s3_client_factory = providers.Factory(
        boto3.client,
        "s3",
        aws_access_key_id=config.aws.access_key_id,
        aws_secret_access_key=config.aws.secret_access_key,
    )

And so on. I think the principle is clear.....

Summary: As you can see, a non-unique interface leads to sub-optimal code, full of extra calls and actions that could have been avoided. Can you bring the Configuration to a consistent behaviour and a unified interface?

[alexted]

@rmk135 Perhaps it should automatically call load() when initialising a config from a source other than dict

[rmk135]

Hi @alexted ,

I think some extra clarification is needed regarding the configuration provider. The idea behind it is that it never returns you a field value while you're working with it in a declarative container. The provider returns a promise to use the value of the config option that you specified when you declared your dependencies.

The statement below looks erroneous to me:

    # But we can't do that here! Because our config object does not contain fields with configuration data 
    # To fill the fields of our object we will have to make one more call:

Thus, you should never call config.load() and config, = config.get_pydantic_settings() during the declarative container definition:

import boto3
from dependency_injector import containers, providers


class Container(containers.DeclarativeContainer):

    config = providers.Configuration()
    
    # These two lines are not supposed to be here:
    config.load() 
    config, = config.get_pydantic_settings()  

    s3_client_factory = providers.Factory(
        ...
        aws_access_key_id=config.aws.access_key_id,  # <----- This will still work without the lines above
        aws_secret_access_key=config.aws.secret_access_key,  # <----- This will still work without the lines above
    )

Also, the interface of the library is consistent -- you define what you need in the declarative phase and then populate the container with data in the runtime phase:

import boto3
from dependency_injector import containers, providers


class Container(containers.DeclarativeContainer):

    config = providers.Configuration()

    s3_client_factory = providers.Factory(
        boto3.client,
        "s3",
        aws_access_key_id=config.aws.access_key_id,
        aws_secret_access_key=config.aws.secret_access_key,
    )


if __name__ == "__main__":
    container = Container()
    
    # Use either of the sources here:
    container.config.from_dict()
    container.config.from_ini("./config.ini")
    container.config.from_yaml("./config.yml")
    container.config.from_json("./config.json")
    container.config.from_pydantic(Settings())
    
    s3_client = container.s3_client_factory()

I hope that makes at least some sense. Maybe you could explain the issues that you face in more detail if not.

[alexted]

@rmk135 Well, in my case everything you wrote above doesn't work. I have web-application based on FastAPI:

di_container.py

from logging.config import dictConfig

from dependency_injector import containers, providers

from src.data.network_clients.idp_facade import IDPRepository
from src.data.yugabytedb import repositories
from src.data.yugabytedb.engine import Database
from src.service.config import AppConfig, CONFIG_FILE
from src.service.logging import get_logging_config
from src.use_cases import countries
from src.use_cases import languages
from src.use_cases import projects
from src.use_cases import users


class Container(containers.DeclarativeContainer):
    config = providers.Configuration(pydantic_settings=[AppConfig(_env_file=CONFIG_FILE, _env_file_encoding="utf-8")])

    logging = providers.Resource(dictConfig, get_logging_config(config)
    # the second version i tried after your post above
    # logging = providers.Resource(dictConfig, get_logging_config(config.LOG_LEVEL, config.APP_NAME, config.ENVIRONMENT))

    db = providers.Singleton(Database, config=config)

    # IDP
    idp_repository = providers.Singleton(
        IDPRepository,
        url=config.IDP_URL,
    )

    # Language
    language_repository = providers.Factory(
        repositories.LanguageRepository,
        connection=db.provided.connection,
    )

    get_languages_use_case = providers.Factory(
        languages.GetLanguagesUseCase,
        languages_repo=language_repository,
    )

    create_language_use_case = providers.Factory(
        languages.CreateLanguageUseCase,
        languages_repo=language_repository,
    )

    update_language_use_case = providers.Factory(
        languages.UpdateLanguageUseCase,
        languages_repo=language_repository,
    )

    delete_language_use_case = providers.Factory(
        languages.DeleteLanguageUseCase,
        languages_repo=language_repository,
    )

    # Country
    country_repo = providers.Factory(
        repositories.CountryRepository,
        connection=db.provided.connection,
    )

    get_countries_use_case = providers.Factory(
        countries.GetCountriesUseCase,
        country_repo=country_repo,
    )

    get_country_by_code_use_case = providers.Factory(
        countries.GetCountryByCodeUseCase,
        country_repo=country_repo,
    )

    create_country_use_case = providers.Factory(
        countries.CreateCountryUseCase,
        country_repo=country_repo,
    )

    update_country_use_case = providers.Factory(
        countries.UpdateCountryUseCase,
        country_repo=country_repo,
    )

    delete_country_use_case = providers.Factory(
        countries.DeleteCountryUseCase,
        country_repo=country_repo,
    )

    # User
    user_repository = providers.Factory(
        repositories.UserRepository,
        connection=db.provided.connection,
    )

    create_user_use_case = providers.Factory(
        users.CreateUserUseCase,
        user_repo=user_repository,
        idp_repo=idp_repository,
        project_repo=project_repo,
    )

    update_user_use_case = providers.Factory(
        users.UpdateUserUseCase,
        user_repo=user_repository,
        idp_repo=idp_repository,
        project_repo=project_repo,
    )

    delete_user_use_case = providers.Factory(
        users.DeleteUserUseCase,
        user_repo=user_repository,
        idp_repo=idp_repository,
        project_repo=project_repo,
    )

    get_user_by_uuid_use_case = providers.Factory(
        users.GetUserByIdUseCase,
        user_repo=user_repository,
    )

    get_users_use_case = providers.Factory(
        users.GetUsersUseCase,
        user_repo=user_repository,
    )

    # Project
    create_project_use_case = providers.Factory(
        projects.CreateProjectUseCase,
        project_repo=project_repo,
        languages_repo=language_repository,
        idp_repo=idp_repository,
    )

    get_projects_use_case = providers.Factory(
        projects.GetProjectsUseCase,
        project_repo=project_repo,
    )

    get_project_by_id_use_case = providers.Factory(
        projects.GetProjectByIdUseCase,
        project_repo=project_repo,
    )

    update_project_use_case = providers.Factory(
        projects.UpdateProjectUseCase,
        project_repo=project_repo,
    )

    delete_project_use_case = providers.Factory(
        projects.DeleteProjectUseCase,
        project_repo=project_repo,
    )

logging.py

import logging
from typing import Dict, Any

from src.service.constants import EnvironmentEnum

from src.service.middleware import request_id_manager


class RequestIdFilter(logging.Filter):
    def filter(self, record) -> bool:  # noqa: A003
        record.request_id = request_id_manager.get()
        return True


def get_logging_config(config) -> Dict:
    logging_config: Dict[str, Any] = {
        "version": 1,
        "disable_existing_loggers": False,
        "filters": {
            "request_id": {
                "()": RequestIdFilter,
            },
        },
        "formatters": {
            "default": {
                "format": "%(levelname)s::%(asctime)s:%(name)s.%(funcName)s:%(request_id)s\n%(message)s\n",
                "datefmt": "%Y-%m-%d %H:%M:%S",
            },
            "json": {
                "format": "%(levelname)s::%(asctime)s:%(name)s.%(funcName)s:%(request_id)s\n%(message)s\n",
                "datefmt": "%Y-%m-%d %H:%M:%S",
                "()": "pythonjsonlogger.jsonlogger.JsonFormatter",
                "json_ensure_ascii": False,
            },
        },
        "handlers": {
            "console": {
                "level": config.LOG_LEVEL,
                "class": "logging.StreamHandler",
                "formatter": "json",
                "stream": "ext://sys.stdout",
                "filters": ["request_id"],
            },
        },
        "loggers": {
            config.APP_NAME: {
                "level": config.LOG_LEVEL,
                "handlers": (["console"]),
            },
        },
    }

    if config.ENVIRONMENT == EnvironmentEnum.LOCAL:
        logging_config["handlers"]["console"]["formatter"] = "default"

    return logging_config

application.py

import sentry_sdk
from fastapi import FastAPI, status
from fastapi.exceptions import RequestValidationError, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import ORJSONResponse, Response
from prometheus_fastapi_instrumentator import Instrumentator
from sentry_sdk.integrations.asgi import SentryAsgiMiddleware
from starlette.middleware.base import BaseHTTPMiddleware

from src.api import v1_routes
from src.service.constants import responses, EnvironmentEnum
from src.service.dependency_injection import Container
from src.service.error_handling import ExceptionsHandler
from src.service.middleware import error_handler, COMMON_ERROR_HANDLERS, error_handling
from src.service.middleware import log_requests
from src.service.middleware import get_or_set_request_id


def create_app() -> FastAPI:
    container = Container()
    app_name = container.config.APP_NAME().lower()
    # ^^^ Here, by the way, is one of the weird things I noticed while 
    # I was debugging  trying to figure out what the hell was going on. 
    # As I wrote above, I tried to get the config in different ways
    # and during my attempts I found that sometimes 
    # I had to do `container.config.APP_NAME`, but other times it didn't work and 
    # I had to refer to the config field as a method `container.config.APP_NAME()`! 
    # And, sometimes, even that didn't work, lol.
    # DI could return nothing, it could return an error that 
    # there is no such attribute, or it could return something obscure 
    # like a memory address - all this didn't help me understand 
    # the behaviour of the lib in literally no way.
    # My brain is exploding.

    app = FastAPI(
        title="core",
        description="The service provides an REST API for manipulating business entities and processes.",
        version="1.2.0",
        default_response_class=ORJSONResponse,
        responses=responses,
        swagger_ui_init_oauth={
            "clientId": app_name,
            "appName": app_name,
            "scopes": ("openid", "email"),
            "usePkceWithAuthorizationCodeGrant": False,
        },
        license_info={"name": "My Proprietary Software License", "url": "https://moc.com/terms"},
    )

    app.container = container

    @app.get("/health", status_code=200, include_in_schema=False)
    def health_check() -> Response:
        return Response(status_code=status.HTTP_200_OK)

    app.include_router(v1_routes, tags=["v1"])

    app.exception_handlers[HTTPException] = error_handler(ExceptionsHandler(*COMMON_ERROR_HANDLERS))
    app.exception_handlers[RequestValidationError] = error_handler(ExceptionsHandler(*COMMON_ERROR_HANDLERS))

    if container.config.ENVIRONMENT() == EnvironmentEnum.PROD:
        sentry_sdk.init(environment=container.config.ENVIRONMENT(), dsn=container.config.SENTRY_URL())
        app.add_middleware(SentryAsgiMiddleware)

    Instrumentator(excluded_handlers=["/health", "/metrics"]).instrument(app).expose(app, include_in_schema=False)

    app.add_middleware(BaseHTTPMiddleware, dispatch=get_or_set_request_id)
    app.add_middleware(
        BaseHTTPMiddleware,
        dispatch=error_handling(ExceptionsHandler(*COMMON_ERROR_HANDLERS)),
    )
    app.add_middleware(BaseHTTPMiddleware, dispatch=log_requests)
    app.add_middleware(
        CORSMiddleware,
        allow_origins=["*"],
        allow_credentials=True,
        allow_methods=["*"],
        allow_headers=["*"],
    )

    return app

In the container, during initialization, I can't initialize any resources/dependencies using config. I can only get the config when the container is already initialized (see application.py). No matter how I tried to do it, nothing works. This is not normal. It's totally counter-intuitive! Especially since the documentation doesn't give any clear information about it.

[rmk135]

@alexted thanks for sharing the code. Yes, I think I understand where the problem comes from. Let me look into your code in detail and get back to you.

[alexted]

@rmk135 If you need the full code, you can use this app template.

[antonio-antuan]

experiencing the same problem

[alexted]

@rmk135, @ZipFile Perhaps it's time to rethink the current implementation of the Configuration provider: simplify it, make it more universal, parameterizable, and flexible, while reducing its interface to the bare minimum. Additionally, ensure that it can be used not only after the container initialization but also during it. What do you think?​

Instead of:

class Container(containers.DeclarativeContainer):
    config = providers.Configuration()
    # or
    config.load()
    # or
    config = config.get_pydantic_settings()
    # or
    db = providers.Singleton(Database, db_url=config.provided.db_url)
    # or 
    container.config.ENVIRONMENT
    # or
    container.config.ENVIRONMENT()
    # ...
    # how many more options to get the data I need? and which of them are working and in what cases?... omg

​Perhaps it's worth adding a mechanism for "eager" or "immediate" initialization of the Configuration provider? I don't need the configuration after the container's initialization; I need it during the process! 90% of dependencies directly rely on the configuration—it's the central stumbling block, the convergence point of all paths!