[BUG] !reset tag does not work on environment keys when multiple files contibute

[mattsouthern]

Description

When using multiple compose files with the -f option, each can define amendments to a service defined in a previous file. When the first two files define environment variables for a service, subsequent files are unable to use the !reset tag to remove these environment key value pairs, per the example at https://docs.docker.com/compose/compose-file/13-merge/#reset-value. If only one file defines environment key value pairs then another is able to reset them as expected. This probably applies to other parts of the service definition but only the environment key was tested.

Steps To Reproduce

With the following files: test1.yml

services:
  app:
    image: myapp
    ports:
      - "8080:80"
    environment:
      FOO: BAR

test2.yml

services:
  app:
    environment:
      ANOTHER: BAR

override.yml

services:
  app:
    image: myapp
    ports: !reset []
    environment:
      ANOTHER: !reset null
      FOO: !reset null

Run docker compose -f test1.yml -f test2.yml -f override.yml config

Resulting in: (directory run in was named tests)

name: tests
services:
  app:
    environment:
      ANOTHER: BAR
      FOO: BAR
    image: myapp
    networks:
      default: null
networks:
  default:
    name: tests_default

Expected result would be:

name: tests
services:
  app:
    image: myapp
    networks:
      default: null
networks:
  default:
    name: tests_default

Compose Version

Docker Compose version v2.27.0

Docker Environment

Client: Docker Engine - Community
 Version:    26.1.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 599
 Server Version: 26.1.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.15.0-88-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.39GiB
 Name: antigluon
 ID: NA5I:MFNI:AKAR:M2X3:LA3S:Y5B3:4NE2:4WAH:2GFN:PHFF:O7MS:JYUQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  gamma.si:5100
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[mattsouthern]

Additionally would be nice if combining !reset and yaml anchors like the following worked: file1.yml

x-common-anchors:
  app_name: &app_name app

services:
  *app_name :
    image: myapp
    ports:
      - "8080:80"
    environment:
      FOO: BAR

override.yml

x-common-anchors:
  app_name: &app_name app

services:
  *app_name :
    image: myapp
    ports: !reset []
    environment:
      FOO: !reset null

current output of docker compose -f file1.yml -f override.yml config (!reset tag not respected):

name: tests
services:
  app:
    environment:
      FOO: BAR
    image: myapp
    networks:
      default: null
    ports:
      - mode: ingress
        target: 80
        published: "8080"
        protocol: tcp
networks:
  default:
    name: tests_default
x-common-anchors:
  app_name: app

[ndeloof]

environment is managed internally using the KEY=VALUE representation during the merge phase, this is why !reset doesn't work here. Not sure how this could be fixed...

would be nice if combining !reset and yaml anchors like the following worked:

anchor is a yaml syntax fature, applied by the low level yaml parser. This takes place before we merge yaml files together, so this is not possible

[hablutzel1]

Then the documentation at https://docs.docker.com/reference/compose-file/merge/#reset-value is wrong, no?

Reset value ... A base compose.yaml file:

services: app: ... environment: FOO: BAR

And a compose.override.yaml file:

services: app: ... environment: FOO: !reset null

[ndeloof]

Had to re-read the original issue description, there's actually something weird as 3 yaml files get merged, the last one using !reset. If I only use 2 of those the documented behavior applies. Let me investigate

[ndeloof]

My initial diagnostic was right: during yaml merge process, envrionment is converted to use the KEY=VALUE syntax (this is the only way we can merge even without interpolation). So after test1.yml has been merged with test2.yml, the list syntax disables ability to apply !reset as the services.app.environment.FOO yaml path never matches a node.

I don't think there's any simple way to fix this with the compose-go architecture

[hablutzel1]

@ndeloof , thanks for checking. It would be nice to have the documentation updated.

[ndeloof]

@hablutzel1 this is a bug, not to be documented but fixed, while I'm not sure "how" yet

[hablutzel1]

That's even better ;)