[BUG] Overriding environment variables with `-e` doesn't work with docker compose up

[contentfree]

Description

The documentation at https://docs.docker.com/compose/environment-variables/set-environment-variables/#substitute-with---env-file states:

You can override specific environment variables from the command line when starting containers. docker compose --env-file .env.dev up -e DATABASE_URL=mysql://new_user:new_password@new_db:3306/new_database

However, attempting to run docker compose up -e FOO=test results in the error: unknown shorthand flag: 'e' in -e

Steps To Reproduce

1. Have a working docker compose set up
2. Attempt to provide an environment variable to up with the -e shorthand syntax: docker compose up -e FOO=test
3. Witness error

Compose Version

v2.23.3

Docker Environment

Client:
 Version:    25.0.3
Server:
 Server Version: 25.0.3

Anything else?

No response

[ndeloof]

That's a documentation issue, up does not allow setting additional environment variables this way, this option is for run compose subcommand. The reason is that up target multiple containers, so there's no simple way to express one want to pass an environment variable to a specific service. Can you please give me more details on your original issue and the need for this ?

[ndeloof]

addressed in https://github.com/docker/docs/pull/19481

[contentfree]

@ndeloof I don't see how one can argue that -e should not apply to up and simultaneously support that an environment file should.

An .env file that contains FOO=test is exactly the same as -e FOO=test. An env file is used with multiple containers; -e should apply the same way (and override values in the env-file). Do you know of an argument that justifies keeping env-file working for up but not the -e flag?

[contentfree]

It's inconsistent as is. Either remove support for env files (no… don't really…) or support -e for docker compose up. I, for one, certainly expected -e to work to override a value in my .env file because that would be consistent behavior.

[ndeloof]

I don't see how one can argue that -e should not apply to up and simultaneously support that an environment file should.

environment file applies to interpolation, not on setting container's environment

An .env file that contains FOO=test is exactly the same as -e FOO=test. An env file is used with multiple containers; -e should apply the same way (and override values in the env-file). Do you know of an argument that justifies keeping env-file working for up but not the -e flag?

No. An .env file that contains FOO=test can be used within compose.yaml for interpolation or to be propagated in container's environment if the definition includes environment: FOO, otherwise it has no impact on container

[ndeloof]

override a value in my .env file

just do:

$ FOO=BAR docker compose up

shell environment have higher precedence setting the project reference variables

[luciangabor]

Perhaps the --env/-e could be added to the main set of compose options. It would allow to easily overwrite a variable, without relying on the shell (and I think Windows's shell/cmd might be the culprit).

It would complement docker compose --env-file <...> and allow customizing individual variables.

[ndeloof]

This would then be very confusing : What would you expect docker compose -e FOO=BAR run -e FOO=BAZ service to do?

[luciangabor]

Well, yes, there's no way around understanding where and how those variables are used.

[contentfree]

An .env file that contains FOO=test can be used within compose.yaml for interpolation or to be propagated in container's environment if the definition includes environment: FOO, otherwise it has no impact on container

I don't see the distinction. One could (& from the genesis of this issue, some do) apply the same "it's for interpolation" argument to the -e flag. If a container in a compose file includes an environment variable and an environment file can be used to interpolate those values, the same applies to the -e flag...

[contentfree]

This would then be very confusing : What would you expect docker compose -e FOO=BAR run -e FOO=BAZ service to do?

Seems straight forward to go with the last provided value.

[ndeloof]

variables in shell environment and .env file are used for interpolation in compose.yaml, they not NOT impact the container environment until you have explicitly configured your compose.yaml definition with some environment entries to do so variable set by docker compose run --env DIRECTLY defined container's environment, and are not involved during compose.yaml parsing

illustration:

$  cat compose.yaml 
services:
  test:
   image: alpine
   command: env
   environment:
    - DEBUG=${FOO}  # set DEBUG container variable according to host's environment FOO
$ FOO=BAR BAR=ZOT docker compose up
[+] Running 1/0
 ✔ Container truc-test-1  Recreated                                                                   0.0s 
Attaching to test-1
test-1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test-1  | HOSTNAME=dcaf1b527538
test-1  | DEBUG=BAR                         
test-1  | HOME=/root
test-1 exited with code 0
## no BAR variable set
$ FOO=BAR docker compose run -e BAR=ZOT test
DEBUG=BAR
BAR=ZOT
## BAR variable set
...

[luciangabor]

While just trying to help (while benefiting from fixes/enhancements that are important to me), and trying to make the point that the ... point of the quoted statement from that documentation page (override specific environment variables from the command line), is about being specific, but it doesn't look like it was ever implemented and it's misleading about being ... about up, I stumbled upon a quirk:

$ compose.yaml() {
    cat << 'EOF'
services:
  service:
    image: ${FOO:?}
EOF
}

(
    set -x
    docker compose --env-file <(echo FOO=BAR) -f <(compose.yaml) config
)
++ echo FOO=BAR
+ docker compose --env-file /dev/fd/63 -f /dev/fd/62 config
++ compose.yaml
++ cat
error while interpolating services.service.image: required variable FOO is missing a value

[contentfree]

variables in shell environment and .env file are used for interpolation in compose.yaml, they not NOT impact the container environment until you have explicitly configured your compose.yaml definition with some environment entries to do so variable set by docker compose run --env DIRECTLY defined container's environment, and are not involved during compose.yaml parsing

OK. I understand the implications for run. But it still seems like -e could apply to up in the same way as shell variables and environment files are used for interpolation.

If I have an .env that has API_KEY in it and my compose yaml uses it for a couple of containers, it seems reasonable that running up -e API_KEY="..." could (should) be treated the same way as changing the value in the .env file.

[ndeloof]

If I have an .env that has API_KEY in it and my compose yaml uses it for a couple of containers, it seems reasonable that running up -e API_KEY="..." could (should) be treated the same way as changing the value in the .env file.

in such circumstances, you don't need a new flag, just run API_KEY=xxx docker compose up as shell environment has precedence on .env file