Feature request: Cascaded settings files

[williwacker]

Hi, I am running Django with multiple tenants. For each tenant I am using a different settings.ini file. But each of this files has common parameters like e.g. DB credentials. In order to only maintain these kind of common parameters only once I would like to have one common-settings.ini file beside the tenant specific settings.ini files. The settings.py is stored in GIT and therefore cannot save credentials. Would it be possible to read more than one settings file? Thanks

[lucasrcezimbra]

I don't think that multiple files make sense for python-decouple. One of the main ideas of the lib is to avoid multiple settings files by reading variables from the environment.

If you really need this multiple files behavior, you can try implementing an extension as described in #115.

[RamonGiovane]

What if I have 3 environments? A development environment, a testing enviroment and a production environment. They have different variables, like different database connection strings, different API keys. Should I place all those variables in the same file, like this?

TEST_DB = 'postgres//user:password@localhost/test
DEV_DB = 'postgres/user:passoword@localhost/dev'
PROD_DB = 'postgres/user:[email protected]/database'

I wish I could have:

DB = config('DB')

And config loads the env from a file accordingly the actual environment. Is it possible?

[lucasrcezimbra]

What if I have 3 environments? A development environment, a testing enviroment and a production environment. They have different variables, like different database connection strings, different API keys. Should I place all those variables in the same file, like this?

TEST_DB = 'postgres//user:password@localhost/test
DEV_DB = 'postgres/user:passoword@localhost/dev'
PROD_DB = 'postgres/user:[email protected]/database'

I wish I could have:

DB = config('DB')

And config loads the env from a file accordingly the actual environment. Is it possible?

In the development environment you will have:

DB = 'postgres//user:password@localhost/dev'

In the testing environment you will have:

DB = 'postgres//user:password@localhost/test'

In the prod environment you will have:

DB = 'postgres/user:[email protected]/database'

[b0o]

Here's what I'm using to cascade multiple repositories:

from decouple import Config, RepositoryEnv, RepositoryEmpty


class RepositoryComp(RepositoryEmpty):
    def __init__(self, *repositories):
        self.repositories = repositories

    def __getitem__(self, key):
        for repository in self.repositories:
            if repository.__contains__(key):
                return repository[key]
        raise KeyError(key)

    def __contains__(self, key):
        for repository in self.repositories:
            if repository.__contains__(key):
                return True
        return False


config = Config(RepositoryComp(RepositoryEnv('.private.env'), RepositoryEnv('.env')))

[henriquebastos]

Hey @b0o. This is cool. I wonder if using a ChainMap wound be enough. I didn't tested, but would be something like:

from collections import ChainMap
from decouple import Config, RepositoryEnv

config = Config(ChainMap(RepositoryEnv(".private.env"), RepositoryEnv(".env")))

[b0o]

@henriquebastos That does work with RepositoryEnv envs but not RepositoryIni envs. For example

private.ini:

[settings]
FOO=Hello

config.ini:

[settings]
BAR=Goodbye

In [1]: from collections import ChainMap
   ...: from decouple import Config, RepositoryIni
   ...:
   ...: config = Config(ChainMap(RepositoryIni("private.ini"), RepositoryIni("config.ini")))

In [2]: config.get("FOO")
Out[2]: 'Hello'

In [3]: config.get("BAR")
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/.asdf/installs/python/3.10.2/lib/python3.10/configparser.py:790, in RawConfigParser.get(self, section, option, raw, vars, fallback)
    789 try:
--> 790     value = d[option]
    791 except KeyError:

File ~/.asdf/installs/python/3.10.2/lib/python3.10/collections/__init__.py:982, in ChainMap.__getitem__(self, key)
    981         pass
--> 982 return self.__missing__(key)

File ~/.asdf/installs/python/3.10.2/lib/python3.10/collections/__init__.py:974, in ChainMap.__missing__(self, key)
    973 def __missing__(self, key):
--> 974     raise KeyError(key)

KeyError: 'bar'

During handling of the above exception, another exception occurred:

NoOptionError                             Traceback (most recent call last)
Cell In[3], line 1
----> 1 config.get("BAR")

File ~/.local/share/virtualenvs/venv-BR9r7xws/lib/python3.10/site-packages/decouple.py:89, in Config.get(self, option, default, cast)
     87     value = os.environ[option]
     88 elif option in self.repository:
---> 89     value = self.repository[option]
     90 else:
     91     if isinstance(default, Undefined):

File ~/.asdf/installs/python/3.10.2/lib/python3.10/collections/__init__.py:979, in ChainMap.__getitem__(self, key)
    977 for mapping in self.maps:
    978     try:
--> 979         return mapping[key]             # can't use 'key in mapping' with defaultdict
    980     except KeyError:
    981         pass

File ~/.local/share/virtualenvs/venv-BR9r7xws/lib/python3.10/site-packages/decouple.py:137, in RepositoryIni.__getitem__(self, key)
    136 def __getitem__(self, key):
--> 137     return self.parser.get(self.SECTION, key)

File ~/.asdf/installs/python/3.10.2/lib/python3.10/configparser.py:793, in RawConfigParser.get(self, section, option, raw, vars, fallback)
    791 except KeyError:
    792     if fallback is _UNSET:
--> 793         raise NoOptionError(option, section)
    794     else:
    795         return fallback

NoOptionError: No option 'bar' in section: 'settings'

If RepositoryIni is updated to except a NoOptionError and re-raise a KeyError, it works:

In [35]: from decouple import RepositoryEmpty, read_config, DEFAULT_ENCODING
    ...: from configparser import ConfigParser, NoOptionError
    ...: class RepositoryIni(RepositoryEmpty):
    ...:     """
    ...:     Retrieves option keys from .ini files.
    ...:     """
    ...:     SECTION = 'settings'
    ...:
    ...:     def __init__(self, source, encoding=DEFAULT_ENCODING):
    ...:         self.parser = ConfigParser()
    ...:         with open(source, encoding=encoding) as file_:
    ...:             read_config(self.parser, file_)
    ...:
    ...:     def __contains__(self, key):
    ...:         return (key in os.environ or
    ...:                 self.parser.has_option(self.SECTION, key))
    ...:
    ...:     def __getitem__(self, key):
    ...:         try:
    ...:             return self.parser.get(self.SECTION, key)
    ...:         except NoOptionError:
    ...:             raise KeyError(key)
    ...:

In [36]: config = Config(ChainMap(RepositoryIni("private.ini"), RepositoryIni("config.ini")))

In [37]: config.get("FOO")
Out[37]: 'Hello'

In [38]: config.get("BAR")
Out[38]: 'Goodbye'

[henriquebastos]

Great work, @b0o! Thank you.

[danrossi]

After multiple attempts this works. Putting the overriding configs before the default configs.

config = Config(ChainMap(RepositoryEnv(".custom.env"), RepositoryEnv(".env")))
print(config('AGENT_PUBLIC_KEY'))