[BUG] environment directive with unset value no longer takes precedence over env_file

[ShadowLNC]

Description

Within the environment directive, if you define a variable without a value, or set it to null (e.g. VAR: ), it is supposed to defer to the shell and unset it in the container if the shell doesn't define it. The unset behaviour no longer overrides a declaration in an env_file.

This behaviour has changed in a recent Compose version and breaks compatibility. It definitely works in Compose 2.13.0 and I think it also worked as recently as 2.23.3.

Steps To Reproduce

docker-compose.yml

```yaml services: mine: image: alpine command: env env_file: fakefile.env environment: AWS_PROFILE: XYZ: null SOMETHING: "aaa" ```

fakefile.env

``` AWS_PROFILE="123" XYZ="456" SOMETHING="456" ```

When running docker compose up and seeing the output of the env command, both AWS_PROFILE and XYZ are populated using Docker Compose v2.26.1-desktop.1.

Previously, those environment variables would be unset. This is the behaviour I expect and rely upon.

According to the Compose Spec:

If the value is not resolved, the variable is unset and is removed from the service container environment.

This is no longer the case. If my shell does set the value, then (and only then) is it overridden.

The precedence table in the docs does not appear to define the behaviour for such a scenario where the shell does not define the variable.

Compose Version

$ docker-compose version
Docker Compose version v2.26.1-desktop.1
$ docker compose version
Docker Compose version v2.26.1-desktop.1

Docker Environment

docker info

Client:
 Version:    26.0.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     /Users/scott/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     /Users/scott/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     /Users/scott/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/scott/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /Users/scott/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/scott/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /Users/scott/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/scott/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     /Users/scott/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/scott/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/scott/.docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 41
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.22-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 7.755GiB
 Name: docker-desktop
 ID: 85b60fc6-eedd-4b8c-996a-89f62a45ab85
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/scott/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Anything else?

I would like to request that there be a mechanism for the environment key to unset a value (not just inherit from the shell) - currently there appears to be no way to override an env_file that I may not have full control over (e.g. dynamically generated).

Ideally, the following behaviour could be implemented:

VARIABLE: null  # unset, or maybe (for backwards compatibility) inherit from the shell only if set
VARIABLE: !reset null  # unconditionally unset, if the above line doesn't unconditionally unset
VARIABLE: "${VARIABLE}" # inherit via interpolation as current, with warning if unset and defaulting to empty string
VARIABLE: "standard value"

[dandn]

I have a similar issue except I was defining an ENV variable (e.g. VAR) in a Dockerfile. As described above, it can no longer be unset by using a corresponding entry (e.g. VAR:) in the docker-compose.yml.

I am using the docker image to run docker compose.

Last working version was in Docker image docker:25.0.3 -> Docker Compose version v2.24.6 Starts failing in Docker image docker:25.0.4 -> Docker Compose version v2.25.0

[lopezator]

Got bitten by this as well, we were setting our DOCKER_TLS_CERTDIR like this:

environment:
   DOCKER_TLS_CERTDIR:

And suddenly all of our builds stopped working. This is a breaking change as was quite hard to debug.

We fixed it with this:

environment:
   DOCKER_TLS_CERTDIR: ""

[dandn]

I can confirm that the release of Docker image docker:27.1.0 (running Docker Compose version v2.29.0) fixed the issue for me.