node
node copied to clipboard
nodejs
`.env` file support issue tracker
This is a follow-up of dotenv support pr to track the development process, and answer some questions that require a general discussion for making the decision to implement them.
Todos:
- [ ] Precisely define the syntax of
.envfiles, and consider a better format not based on ini - [x] Supporting absolute paths for .env file https://github.com/nodejs/node/pull/51425
- [x] Should we support multiple
--env-filethrough the CLI? (cc @GeoffreyBooth) https://github.com/nodejs/node/pull/49542 - [x] Support multi-line values https://github.com/nodejs/node/pull/51289
- [x] Follow up after @tniessen's PR has landed to main https://github.com/nodejs/node/pull/49213
- [x] Should we add a programmatic API? If yes, what would be the use-case? (cc @cjihrig) https://github.com/nodejs/node/pull/51476
Questions:
- [x] ~~Should we merge NODE_OPTIONS if both environment variable and .env version exist? Currently, we prefer environment variable over .env configuration.~~ (cc @ljharb) https://github.com/nodejs/node/pull/49217
- [x] Should we throw if
.envfile does not exist? We currently follow the default behavior ofdotenvpackage and do not throw. (cc @MoLow)
Please feel free to edit this issue in case there are more questions that needs to be answered.
cc @nodejs/tsc
I vote "throw on non-existing env file" and "throw on multiple --env-file parameters" since those are easy to change later and lets us land without waiting for a decision on both.
IMO:
Should we merge NODE_OPTIONS if both environment variable and .env version exist?
The usual practice (of environment variable precedence ordering) is command line overriding stored setting in files. That way, the end user who runs the code has the last say on the settings.
Should we support multiple --env-file through the CLI?
Yes. This will help co-existence of application and its dependencies with their own env definitions.
Should we add a programmatic API?
Yes. This will help embedded use cases where node.js is not started through the regular launcher with regular command line parsing sequences.
Should we throw if .env file does not exist?
Yes. This will help the feature function to be deterministic.
Should we add a programmatic API?
Yes. This will help embedded use cases where node.js is not started through the regular launcher with regular command line parsing sequences.
I assume that this refers to a C++ API for embedders. This might be useful if the API can be used before any of Node.js or any of its dependencies are initialized. Otherwise, embedders will run into the same problem as we did with NODE_OPTIONS.
I assume that this refers to a C++ API for embedders.
Yes, I meant that.
This might be useful if the API can be used before any of Node.js or any of its dependencies are initialized. Otherwise, embedders will run into the same problem as we did with NODE_OPTIONS.
can you please elaborate on the problem we did with NODE_OPTIONS? sorry, I am not up-to-date with that.
What I mean is that environment variables that affect Node.js, its dependenices, the embedding application, or the embedder's dependencies must be set before the respective component attempts to use it. In many cases, that means before initialization of the respective component.
(On the other hand, reading environment variables from a file is simple enough, so if an application doesn't use the Node.js CLI, it probably might as well implement this feature itself.)
What I mean is that environment variables that affect Node.js, its dependenices, the embedding application, or the embedder’s dependencies must be set before the respective component attempts to use it. In many cases, that means before initialization of the respective component.
Yes. As in, it doesn’t do much good to write JavaScript code like process.env.NODE_OPTIONS = '...' because by the time this JavaScript code runs, Node has already loaded and configured itself and so most (all?) of what you would set by this point is too late to have any effect.
Re merging versus overwriting, I feel somewhat strongly that we should overwrite, because this .env file support is general purpose for any environment variable and not just Node-specific ones, and we can’t know how to merge variables like DATABASE_PASSWORD or whatever. I think it would be confusing UX if Node environment variables got merged but non-Node ones didn’t. I was assuming that whatever variables were set in the environment would take precedence over ones set in a loaded file, but if others feel strongly that it should be the reverse I don’t have a strong opinion on this part.
I would assume that individual variables would overwrite but that if the env has A and the file has B that the process would end up with both A and B.
I would assume that individual variables would overwrite but that if the env has A and the file has B that the process would end up with both A and B.
@ljharb This is the default behavior right now, but do not merge NODE_OPTIONS.
➜ node git:(dotenv-support) cat .env
TESTING_KEY=THIS_IS_VALUE
➜ node git:(dotenv-support) TESTING_KEY=OVERRIDEN ./out/Release/node -e "console.log(process.env.TESTING_KEY)"
OVERRIDEN
ah ok, gotcha. then yeah i think https://github.com/nodejs/node/issues/49148#issuecomment-1677741025 makes sense because indeed it would get confusing figuring out which node options are mutually exclusive, which are single values, which are multiple values, etc.
I was wondering, do you plan to support also something like dotenv-vault that lets you encrypt your secret and decrypt it just in time.
I was wondering, do you plan to support also something like
dotenv-vaultthat lets you encrypt your secret and decrypt it just in time.
Not at the moment, but contributions are welcome.
I was just seeing how this feature worked and I think that either the documentation or the feature is wrong in the 20.6.0 release.
If the same variable is defined in the environment and in the file, the value from the environment takes precedence.
But, I checked out the v20.6.0-proposal branch, built the latest executable and tried the following:
ϟ echo $PHIL_VAR
what
node (4e4bb9f)
ϟ ./node --version
v20.6.0
node (4e4bb9f)
ϟ ./node -e "console.log(process.env.PHIL_VAR)"
what
node (4e4bb9f)
ϟ ./node -e "console.log(process.env.PHIL_VAR)" --env-file .env
hello
node (4e4bb9f)
ϟ cat .env
PHIL_VAR=hello
From the documentation, I would have expected the PHIL_VAR variable set in the environment to always be the result. Instead, it seems that the .env file version of the PHIL_VAR variable over-rides it when it is used.
Does this need a quick documentation fix ("the value from the .env file takes precedence") or does the feature need fixing?
does the feature need fixing?
The feature needs fixing. The environment should take precedence over the file.
The environment should take precedence over the file.
I don't think so. The whole point of env files is to override the local environment.
The environment should take precedence over the file.
I don't think so. The whole point of env files is to override the local environment.
We've discussed this quite a bit in this pull request, but every existing implementation of dotenv, including dotenv-node and bun, does not override the environment by default. They tend to have an option to override, which we should consider implementing, but to make overriding the default would go against every existing implementation.
@anonrig Big and important use-case for programmatic API is setup hooks for testing frameworks, such as vitest or jest.
I opened a PR to address multiple --env-file declarations support: https://github.com/nodejs/node/pull/49542
@anonrig Big and important use-case for programmatic API is setup hooks for testing frameworks, such as vitest or jest.
Thanks @kibertoad. It is on my agenda.
Big and important use-case for programmatic API is setup hooks for testing frameworks, such as vitest or jest.
@kibertoad Previously, the programmatic API was described as a C++ API for embedders. Is that what you have in mind? If not, could you clarify?
@tniessen I mean being able to put an equivalent of require('dotenv').config() inside a hook :D
In that case, this discussion has already requested two very different runtime APIs. The JS runtime API likely does not need to live in core since it won't be able to properly set NODE_OPTIONS etc.
@tniessen That would be an understandable limitation of JS API. But if native .env support can't be used from JS userspace, then users will still need to depend on dotenv for tests, and then there are no good reasons not to use dotenv for everything else as well, for consistency.
Precisely define the syntax of .env files, and consider a better format not based on ini
I think for this, we can just use existing formats like YAML or TOML, that way we don't have to redefine any conventions and just follow the standard definitions provided by those common formats. A breaking change, but certainly handles scenarios like multi-line values and other issues that are present with the INI format environment file.
If we have to change the format, I would go with TOML.
Now, this change leads to a huge breaking change from all community packages (even cross-language/runtime).
What I imagined for that TODO was that we support a JSON file as an additional config file that was enabled via a different flag, like --config=node.config.json. Node already parses JSON config files such as package.json and policies.
What I imagined for that TODO was that we support a JSON file as an additional config file that was enabled via a different flag, like
--config=node.config.json. Node already parses JSON config files such aspackage.jsonand policies.
Yes, JSON is also good and it is baked into Node, but was thinking that the other two formats has better multi-line/line-break support, which is a common occurrence in using .env files.
In any case, this TODO looks promising and I am looking forward for its feature complete implementation.
It's true that JSON is easier to support, but it's not a good format for humans to edit.
I think what should have happened is a precise specification of the file format, and perhaps a reconsideration of the format before the feature was released.
YAML, JSON, and TOML all seem rather complicated for something as trivial as a flat list of environment variables. More complicated configuration files are out of scope for this feature (and I am not convinced that they are desirable at all).
I agree with @tniessen that discussion about of an additional config file shouldn't be related to the .env feature.
ini seems pretty simple, and since npmrc uses ini, if node added a builtin ini parser/serializer, that seems like a major benefit to the ecosystem regardless.