feat: support reading/loading multiple .env files

[duarte-pompeu]

This PR allows reading from multiple .env files in the dotenv CLI, specifically the targets get, list and run.

This will allow users to have a big .env file with default configs, and smaller ones which only incremental changes - therefore removing the burden of repeating unchanged key-values.

The goal for the new behavior is as follows:

• files are processed in order
• new keys lead to creating new key-values
• repeated keys lead to overriding previous values

Extra remarks:

• this PR preserves the behavior of working with single files, and falling back to .env when no file is specified
• this PR forbids write operations on multiple files, specifically set and unset

Example:

$ echo -e 'URL="https://www.example.com"\nUSER="user"\nPASSWORD="secret"' > .env1
$ echo -e 'USER="admin"\nPASSWORD="admin"' > .env2
$ python -m dotenv -f .env1 -f .env2 list
PASSWORD=admin
URL=https://www.example.com
USER=admin

Ideas for the future:

• also allow loading multiple .env files from the code, eg load_dotenv(f1, f2, ...)
• also allow setting and unsetting values for multiple .env files

Relates to #418 .

[pheanex]

@duarte-pompeu will you take care of resolving the conflicts before we ping the maintainer for review?

[duarte-pompeu]

I can give it a try tonight, it would be interesting to merge this feature.

[duarte-pompeu]

Solved conflicts but there's still some work to do regarding failing tests.

[duarte-pompeu]

Think it looks better now @pheanex 🙂

It's missing more tests and documentation, but I'd prefer to have feedback from a maintainer before tackling those.

[pheanex]

Awesome! Let's see if @bbc2 or @theskumar is available for a quick review?

[theskumar]

Fundamentally as feature I'm good with supporting this. I like your approach and seeing how you are going about it. I haven't had time to look into the implementation details. But I feel free to improve on your implementation. Would really appreciate keeping it backward compatible. Thank you @duarte-pompeu for your contribution.

[duarte-pompeu]

Thanks for the feedback @theskumar. I can keep working on this over the next days.

I'm thinking about reducing the scope in this PR though - would it be OK to only support read operations, such as get and list? Seems like a useful increment and later we can think about how to handle write operations.

[theskumar]

Read is completely fine, I could not see how the write be extended to multiple files. Would be curious if you have some ideas around it.

[duarte-pompeu]

Read is completely fine, I could not see how the write be extended to multiple files. Would be curious if you have some ideas around it.

Actually I already implemented it: unset looks for the key in all files and deletes it when present. But I have some doubts about this usefulness, or if this behavior is intuitive to users - as such, I'd rather remove it from this PR and discuss it later.

Example as of 49c34a5:

$ for f in $(ls .env*); do echo $f; cat $f; echo; done
.env1
a=1
b=2
.env2
b=2
c=3
# unset 'a', which is only present in .env1
$ python -m dotenv -f .env1 -f .env2 unset a
Successfully removed a from .env1.
$ for f in $(ls .env*); do echo $f; cat $f; echo; done
.env1
b=2
.env2
b=2
c=3
# unset 'b', which is present in both .env1 and .env2
$ python -m dotenv -f .env1 -f .env2 unset b
Successfully removed b from ['.env1', '.env2'].
$ for f in (ls .env*); do echo $f; cat $f; echo; done
.env1
.env2
c=3
# try to unset key which does not exist
$ python -m dotenv -f .env1 -f .env2 unset d
Key d not removed from ('.env1', '.env2') - key doesn't exist.

[duarte-pompeu]

I think this is ready for review @theskumar:

• implemented changes
• kept backward compatibility (I think so, someone please double check for peace of mind 😃)
• added tests
• updated docs
• improved PR description

We didn't talk about this, but I only implemented this for the CLI utility. If there's interest to also enable this behavior from the library code, I'm up to contribute to it (but would prefer to have it in a separate PR, as I find this one already useful as is).

[duarte-pompeu]

Hi @theskumar and @pheanex , any feedback?

[pheanex]

Lgtm. Looking forward to this ☺️👍

[theskumar]

@duarte-pompeu thanks for all the hard work. Unfortunately, I've had not been able to review this yet. Looks like we Christmas 🎄 break would be a good time to taking care of it for me, along with few more pull requests.

[duarte-pompeu]

@duarte-pompeu thanks for all the hard work. Unfortunately, I've had not been able to review this yet. Looks like we Christmas 🎄 break would be a good time to taking care of it for me, along with few more pull requests.

Cool, I also have some free time now so it would also work for me 😃

[duarte-pompeu]

Hi @theskumar , I agree with your feedback and addressed in the last commits. What do you think?

[pheanex]

@theskumar any chance to take a quick look again?

[pheanex]

@bbc2 any chance you can take a look at this? It seems @theskumar is currently unavailable.

[pheanex]

@bbc2 as two more weeks have passed: Are we good to move forward with this?