[BUG] Variable interpolation is not working except for .env

[clementgineste]

Description

Any interpolation variables from something else than .env is not working; using env_file and interpolating using environment

As discussed in https://github.com/docker/compose/issues/11741

Steps To Reproduce

1. Use this compose.yaml

services:
  bug-example:
    image: alpine:latest
    container_name: container-example
    command: sleep 1000
    env_file:
      - example.env
    environment:
      - HOST_VAR_ONE=${HOST_EXAMPLE_ENV:-Default}

2. With this config

example.env

HOST_EXAMPLE_ENV=InterpolatedVar

.env

RANDOM_VAR=Randomized

3. Run docker compose up --build -d
4. Check that HOST_VAR_ONE is using Default and not InterpolatedVar. `docker exec container-example printenv | grep -i var
5. If you transfert the variable into .env interpolation will work.

Compose Version

clement@fedora:~/work/temp/docker-compose/bug_compose$ docker compose version
Docker Compose version v2.26.1-desktop.1
clement@fedora:~/work/temp/docker-compose/bug_compose$ docker-compose version
Docker Compose version v2.26.1

Docker Environment

docker info

clement@fedora:~/work/temp/docker-compose/bug_compose$ docker info
Client: Docker Engine - Community
 Version:    26.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     /usr/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     /usr/lib/docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /usr/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /usr/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /usr/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /usr/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     /usr/lib/docker/cli-plugins/docker-scout

Server:
 Containers: 18
  Running: 18
  Paused: 0
  Stopped: 0
 Images: 32
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.22-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.442GiB
 Name: docker-desktop
 ID: 7fdd91d8-1880-4324-885c-0d28875b2e73
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///home/clement/.docker/desktop/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

### Tasks

[RiverHeart]

While it would be fantastic if variable interpolation applied to the env_file I don't believe it does. I say that after spending a good deal of time working with Docker Swarm, which doesn't read from .env automatically so env_file needs to be explicitly defined.

https://docs.docker.com/compose/compose-file/compose-file-v3/#env_file

The value of VAL is used as is and not modified at all. For example if the value is surrounded by quotes (as is often the case of shell variables), the quotes are included in the value passed to Compose.

This is made a bit confusing because priority is given to environment vars

Environment variables declared in the environment section override these values – this holds true even if those values are empty or undefined.

So you'll define your environment vars and do a FOO=${FOO} but $FOO is being sourced from your shell, not the env_file and the environment vars substitute to null values which then override the good values in the env_file. You can test this out by running the following command to preview your config file.

sudo HOST_EXAMPLE_ENV=foobar docker compose config | grep HOST

HOST_EXAMPLE_ENV should show up because it's getting sourced from your shell this time. docker compose is just being helpful and sourcing the .env into your shell before performing the interpolation.

Options:

1. Source the env file and export the vars before running compose/deploy
2. Put all our env vars in the env file including the default values which you override as needed and omit the environment block from the compose file.
3. Use a template or structured transform application to perform the substitutions on the compose file before running compose/deploy
4. ?

[nikita-sheremet-java-developer]

Consider another example:

services:
  bug-example:
    image: alpine:${MY_VERSION}
    container_name: container-example
    command: sleep 1000
    env_file:
      - .env.local
      - .env

If .env contains MY_VERSION=latest is works when it missing - it fails. No matter has .env.local definition for MY_VERSION ot not.

I mean that bug exists even when environment do not present

[ndeloof]

env_file and environment are not used for interpolation, but to set container's environment. .env file is the default source for interpolation, this is why you observe this behavior (can override using --env-file)

this is highly confusing, I've proposed https://github.com/docker/docs/pull/19481 to help clarify this

[clementgineste]

If you use env_file in the include section, then it acts as source(s) for interpolation :D

[ndeloof]

indeed, as a yaml flavor of the equivalent docker compose --env-file flag. The main issue here is that both docker run and docker compose define this same flag, but for a distinct usage