[BUG] Environment Variable handling is maddeningly inconsistent

[ilsaloving]

Description

According to https://docs.docker.com/compose/compose-file/12-interpolation/:

Values in a Compose file can be set by variables and interpolated at runtime. Compose files use a Bash-like syntax ${VARIABLE}. Both $VARIABLE and ${VARIABLE} syntax is supported.

In practice, this is untrue. Variable interpolation is all over the map. Sometimes they work, sometimes they don't.
Different settings handle interpolation differently. (eg: https://github.com/docker/compose/issues/4249, or ports) Different files handle interpolation differently. (eg: https://github.com/docker/compose/issues/11755)

This is causing needless confusion and frustration, and wastes countless hours of people's time with needless trial and error.

Variables should work across the board. If you're reading variables, regardless of source, variable interpolation should occur. If you're applying settings, regardless of the type, variable interpolation should occur. Heck, if you want to have an entire docker-compose yaml stanza pulled from a variable, do variable interpolation.

Steps To Reproduce

No response

Compose Version

2.24.7

Docker Environment

Client:
 Version:    25.0.4-rd
 Context:    rancher-desktop
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.0
    Path:     /Users/ilsa.loving/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.7
    Path:     /Users/ilsa.loving/.docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 9
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 4e1fe7492b9df85914c389d1f15a3ceedbb280ac
 runc version: 18a0cb0f32bcac2ecc9a10f327d282759c144dab
 init version:
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.14-0-virt
 Operating System: Alpine Linux v3.19
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 5.784GiB
 Name: lima-rancher-desktop
 ID: c8f6b4ee-3832-4f33-85c9-19102fd7f961
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[ndeloof]

interpolation works for sure, in most cases confusion comes from setting values based on env file, and how those get loaded with some default paths.

Different settings handle interpolation differently. (eg: https://github.com/docker/compose/issues/4249, or ports)

actually, this is the opposite. all attribute handle interpolation the same way, and when it comes to list of values to be set, this makes things harder for user as this assumes a fixed number of variables.

Different files handle interpolation differently. (eg: https://github.com/docker/compose/issues/11755)

This one is (yet another) case where user get confused between interpolation and container environment being set by env_file. We get such issue reported every week, so I've proposed https://github.com/docker/docs/pull/19481 as clarification as there's massive confusion in this area with setting container environment.

If you have a specific bug to report, please give more details. I only can read "I'm frustrated this didn't worked out of the box" which won't bring us anywhere. I acknowledge the docs are bad documented this feature and need some love

[ilsaloving]

How the heck do you expect me to file a "specific bug report" when there is no way of knowing how this thing is supposed to be working in the first place? I did just file a report, as have countless other people, and Dockers' developers response is "You just don't know how to use it. Oh and the docs are crap so HAHA sucks to be you."

The only thing that works reliably is if I directly set, or pull an OS environment variable from the immediate environment, in the docker-compose.yaml file. eg:

services:
  my-service:
    environment:
      MY_VAR: my_val
      MY_VAR2: $MY_VAR2
      MY_VAR3: ${MY_VAR3:-default_val}

With external environment files, the only thing I have been able to get working semi-reliably is MY_VAR=my_val. The other two syntaxes do not work in a reliable/predictable fashion.

And heaven forbid I include: a separate docker-compose.yaml file, docker-compose-override.yaml file, or multiple environment files, cause then only $DEITY knows what if anything is supposed to happen. I have a project where I am including one docker-compose file into another, and the environment files specified by the included docker-compose file are ignored entirely.

I don't know or care if it's a code issue or a documentation issue. It is not only a bug, but a CRITICAL bug.

I appreciate you've attempted to get some progress on this, but clearly none of the other developers consider this a problem because https://github.com/docker/docs/pull/19481 has been stalled for months.

If variable handling /w docker compose is so challenging that it cannot even be documented properly, it should be ripped out of the product entirely. As things stand, given Docker's reach, this has probably cost millions in lost productivity. Maybe Docker should spend less time coming up with ways to squeeze it's user base for money, and more time making a product it's users would be willing to pay money for.

[ndeloof]

The other two syntaxes do not work in a reliable/predictable fashion.

Please provide an illustration example so I can confirm this is a bug or explain / propose docs improvements accordingly

the environment files specified by the included docker-compose file are ignored entirely

same request. Please provide an illustration example if this sounds like a bug

none of the other developers consider this a problem

I can understand some frustration, but please also adopt a respectful tone. The team is working hard to maintain Compose with thousands users and we don't have much bandwidth to answer everyone. If you want some assistance to better understand how Compose works, or get your issue considered and fixed, just consider we also have to manage our priorities and a well reported issue with an example we can test and adopt as a new test case will bring faster answer/fix vs just complaining.

[roverwolf]

There is definitely inconsistency with the ways that environment variable interpolation happens, some due to confusion based on the similar naming of --env-file and env_file, and also based on interpreting the documentation in an "obvious" way, but I guess not intended way.

For example, the Additional information section of https://docs.docker.com/compose/environment-variables/set-environment-variables/#use-the-env_file-attribute states

"In addition, as the .env file supports interpolation, it is possible to combine those with values set by environment."

but this does not appear to be true of a .env file added using the env_file attribute which this section is about, only for .env files passed via --env-file (or the default .env file).

Similarly, https://docs.docker.com/compose/environment-variables/envvars/ indicates that one should be able to override COMPOSE_ENV_FILES using the first two methods listed

• Within your Compose file using the environment attribute
• With the env-file attribute and an environment file

but that also does not appear to be possible (understandable, but incorrectly documented).

[pjlsergeant]

@roverwolf Having the same issue as you with that documentation, I've pulled it out into its own issue as needing a doc patch. I'd rather the underlying functionality was changed, but, the docs being fixed is a good start.

[yakobe]

Is this also a problem when trying to use a variable for ports? For example, with the following service definition:

services:
  web:
    env_file:
      - path: .env
        required: true
      - path: .env.local
        required: false
    ports:
      - "$WEB_PORT:80"

Here i can set WEB_PORT in the .env file, but it is ignored if i set it in the .env.local file.

I've been going insane trying to get it to work? Is it just currently not possible 😅?

[ndeloof]

@yakobe env_file is not used to process compose.yaml and interpolate variables, but to set containers environment. see https://docs.docker.com/compose/how-tos/environment-variables/set-environment-variables/ which isn't the same as https://docs.docker.com/compose/how-tos/environment-variables/variable-interpolation/

I'm closing this issue as there's no expected resolution. confusion between env_file and use of .env as default source for interpolation is a legacy we can't fix

[yakobe]

Wow that is confusing. So I can control it with --env-file on the command line but not with the env_file in the compose file?

So is there no built in way to achieve what I'm looking for? The .env file is managed by git, so the only option is to change the values and be careful not to commit it?

[roverwolf]

@yakobe

Wow that is confusing. So I can control it with --env-file on the command line but not with the env_file in the compose file?

Yes, it would be really nice if the documentation would make it explicitly clear that env_file is not the same as --env-file. Besides overriding for local config, another use case I have is to be able to specify a different filename as .env is not unique to docker compose but is interpreted differently (quote rules, etc.) by other tools.

So is there no built in way to achieve what I'm looking for? The .env file is managed by git, so the only option is to change the values and be careful not to commit it?

Not easily, but one (convoluted) way is to use a different directory with a .env file that is like your .env.local file and then set COMPOSE_FILE to point back to the compose file in the original directory. You would then run docker compose in the new directory and that local .env file takes precedence over the .env file beside the compose file. https://docs.docker.com/compose/how-tos/environment-variables/variable-interpolation/#local-env-file-versus-project-directory-env-file

[yakobe]

another use case I have is to be able to specify a different filename as .env is not unique to docker compose but is interpreted differently (quote rules, etc.) by other tools.

Yes, that's the same problem I have. Both docker and the app are competing to be the one that owns the .env file.

Interesting work around 😂. I guess it could work, although a bit different to onboard other team members. Hmmmm 😕