Multiple --env-file cli arguments not be merged correctly, if it's an environment or args configuration option

[LantosIstvan]

Description

This problem came to my attention when I try to spin up a new Laradock project. I thought for a moment I could use multiple environment files to input the docker-compose command:

 $ docker-compose --env-file .env --env-file laradock/.env.example -f laradock/docker-compose.yml -f docker-compose.override.yml config > config2.yml

The environment variables will be merged, except when the environment variable used as an environment: or args: configuration option:

    php-fpm:
      build:
        context: ./php-fpm
        # This won't be merged, referencing `laradock/.env.example` file
        args:
          - LARADOCK_PHP_VERSION=${PHP_VERSION}
        # This will be merged as expected, referencing `.env` file
        volumes:
        - ./php-fpm/php${PHP_VERSION}.ini:/usr/local/etc/php/php.ini

So the generated configuration will look like this (pseudo-code):

    php-fpm:
      build:
        context: ./php-fpm
        args:
          - LARADOCK_PHP_VERSION=7.4
        volumes:
        - ./php-fpm/php8.1.ini:/usr/local/etc/php/php.ini

But this is the expected output, that I try to achieve:

    php-fpm:
      build:
        context: ./php-fpm
        args:
          - LARADOCK_PHP_VERSION=8.1
        volumes:
        - ./php-fpm/php8.1.ini:/usr/local/etc/php/php.ini

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker compose version:

Docker Compose version v2.2.3

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.7.1)
  compose: Docker Compose (Docker Inc., v2.2.3)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.60.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.59GiB
 Name: docker-desktop
 ID: L3FW:H733:UK4X:DDSR:L2GQ:IXOT:OZKA:SOCZ:BH4O:JU7L:R27N:A77A
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional environment details:

$ systeminfo

OS Name:                   Microsoft Windows 11 Pro
OS Version:                10.0.22000 N/A Build 22000

[LantosIstvan]

If env_file in docker-compose.override.yml added as an empty array:

env_file: []

Then why on earth it's not rewriting the array from the original compose file? I understand why the newly added env_file fields appended to the array, but you should give an option to make the entire array empty.

[LantosIstvan]

I should note that, if env_file: .env in the base compose config is a non-existing file, this halts the configuration merging process with docker-compose.override.yml. This is my use case.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[aschleifer]

Are multiple --env-file definitions supported? I tried it and for me the latest definition overrode all previous ones.

[LantosIstvan]

This must be the case, I ended up simplifying my stack. Still, this would be an useful feature. If the YAML files are merged, one can expect the same for .env files.

[aschleifer]

Yes, I would also see this as a supported feature that you can define multiple --env-file and they get merged from first to last definition. So that later definitions override variables if they are defined multiple times.

[LantosIstvan]

Correction: I not simplified my stack, I use a shell script wich merging the .env files with system environment variables, plus adding one with --env-file flag (.env.example from Laradock), a.k.a this means a very dangerous current shell. This shell script magic wouldn't be neccessary, if one can define multiple .env files for the compose command.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Segaja]

This is still a valid issue and should be handled.

[ndeloof]

Can you please confirm this issue is still relevant with latest release (v2.18.1) ? I guess this is already fixed by https://github.com/compose-spec/compose-go/pull/403

[Segaja]

Can you please confirm this issue is still relevant with latest release (v2.18.1) ? I guess this is already fixed by compose-spec/compose-go#403

Yes, I can confirm that https://github.com/compose-spec/compose-go/pull/403 fixes this. This was relased in https://github.com/compose-spec/compose-go/releases/tag/v1.13.5 which in turn was released for docker/compose with https://github.com/docker/compose/releases/tag/v2.18.0

[stale[bot]]

This issue has been automatically marked as not stale anymore due to the recent activity.