feature request: load multiple `.env` file before running a command

[antoine-gallix]

I have different configuration for my application that I intend to be composable. For example imagine I have env files with config values for database server hosts, and other env files with user credentials. I want to choose a combination of the two. I try to do something like

dotenv -f host_config.env -f user_credentials.env run bash do_things_with_db.sh

However it seems that only the last file is taken into account. I think this would be a useful feature to be able to load multiple files. What do you people think about it?

[duarte-pompeu]

I'm also interested in this.

I see two use cases where this is useful:

• a .env file growing too much and dev wants to split in more
• dev wants to change only a few values, for example local / development / staging / production: it can be easier to maintain a base .env file and then overload as needed with local.env/dev.env/etc.

I gave it a try and have a working proof of concept in #424 . It's not complete, but I can work on it if there's interest.

[cbensimon]

@duarte-pompeu Thanks for your work ! Definitely interested by this feature

I can contribute if needed, let me know

[duarte-pompeu]

FYI I was able to find a workaround. Given the following files:

# test1.env
URL="https://www.example.com"
USER="user"
PASSWORD="secret"

# test2.env
USER="admin"
PASSWORD="admin"

# test.py
import os
print(os.environ["URL"])
print(os.environ["USER"])
print(os.environ["PASSWORD"])

We can combine them by chaining multiple calls to dotenv:

$ dotenv -f src/test1.env run -- dotenv -f src/test2.env run -- python src/test.py 
https://www.example.com
admin
admin

[duarte-pompeu]

@duarte-pompeu Thanks for your work ! Definitely interested by this feature

I can contribute if needed, let me know

Great!

I was having some trouble with multiple warnings from get_key when some files don't have the key, but I was able to fix it with https://github.com/theskumar/python-dotenv/pull/424/commits/f730c6d92900f8741ae03f24e177b1a3ef32ecfc .

What do you think about this @theskumar ?

[iuriguilherme]

I'm not sure if my use case is remotely similar to what is being discussed here, but I have something like a list of users and password combination. The actual data is much more complex than that, but the current way I can achieve what I want is defining dictionaries in many package files then importing them:

# user1.py
USER = "user"
PASSWORD = "secret"

# user2.py
USER = "admin"
PASSWORD = "admin"

# user3.py
USER = "anotherone"
PASSWORD = "hackme"

# test.py
from importlib import import_module
allowed_users = ['user1', 'user3']
[import_module(user) for user in allowed_users]
[print(f"credentials: USER={getattr(user, 'USER')}, PASSWORD={getattr(user, 'PASSWORD')}") for user in allowed_users]

expected output:

credentials: USER=user, PASSWORD=password
credentials: USER=anotherone, PASSWORD=hackme

Moral of the story: I want to be able to read variables from several files, and I want to be able to select which ones at each run. But I'm not supposed to overwrite those variables with the last read file. I want instead to append every file to a "list of envs".

Perhaps this is out of scope but that is my requirement.

[duarte-pompeu]

Moral of the story: I want to be able to read variables from several files, and I want to be able to select which ones at each run. But I'm not supposed to overwrite those variables with the last read file. I want instead to append every file to a "list of envs".

@iuriguilherme: I think that's a different scenario: in this issue we want to override values, not use multiple ones.

Anyawy, I'll try to help you. Would this work?

Files:

# user1.env
username="user_1"
password="pass_1"

# user2.env
username="user_2"
password="pass_2"

Python REPL:

>>> from dotenv import *
>>> users_dicts = [dotenv_values(env) for env in ["user1.env", "user2.env"]]
>>> users_dicts
[OrderedDict([('username', 'user_1'), ('password', 'pass_1')]), OrderedDict([('username', 'user_2'), ('password', 'pass_2')])]

>>> from dataclasses import *
>>> @dataclass
... class User:
...     username: str
...     password: str
... 
>>> users
[User(username='user_1', password='pass_1'), User(username='user_2', password='pass_2')]
>>> 
>>> users = [User(**user_dict) for user_dict in users_dicts]
>>> users[0].username
'user_1'
>>> users[1].username
'user_2'

You may also want to take a look at pydantic, which is more advanced than python's dataclasses: https://pydantic-docs.helpmanual.io/

[iuriguilherme]

That's exactly what I'm trying to do, use pydantic BaseSettings in conjunction with dotenv. Because there are scenarios where I can't import configuration files as modules or packages, and it's more reliable to use dotenv files, configparser, etc.

Example of a case that my software breaks if the user fail to provide appropriate configuration files and there is no easy way to debug what's wrong is pip install -e .

[pheanex]

@duarte-pompeu will you still carry this forward or can/should someone else help/take over?

[duarte-pompeu]

I attempted to implement this on #424 but didn't get any feedback. If a maintainer is interested in merging this feature, I can fix the merge conflicts.