Consistency problem with `--env-file` usage at least on `up` and `down` commands

[fabrykato]

Hey guys, connected to docker/compose-cli#1892, I just recently detected a problem with consistency of --env-file usage at least on up and down commands.

Description In our case, we rely on override technique (normally utilized by --project-directory options when running locally). Also we depend a lot on environment variables defined in .env (default) or ci/ci.env (set with --env-file then). Env files and docker-compose ymls are located in a subfolder under project root directory.

Because the --project-directory option did not recognize env files in any way with 2.0.0-beta.6, I tried to roll out to using -f multiple times connected with explicit --env-file. The strange problem with this is as follows:

Steps to reproduce the issue:

1. Run docker-compose -p <project_name> -f ci/docker-compose.yml -f ci/docker-compose.override.yml --env-file ci/.env up -d --build db [+] Building 0.4s (8/8) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 32B 0.0s => [internal] load .dockerignore

... works as expected: builds a custom database image and starts container from it.

2. Run docker-compose -p <project_name> -f ci/docker-compose.yml -f ci/docker-compose.override.yml --env-file ci/.env down time="2021-07-15T15:41:24+02:00" level=warning msg="The NECESSARY_ENVVAR_NAME1 variable is not set. Defaulting to a blank string." time="2021-07-15T15:41:24+02:00" level=warning msg="The NECESSARY_ENVVAR_NAME2 variable is not set. Defaulting to a blank string." ...

... does not interpret the same command options on down command due to the missing env-file contents.

Output of docker-compose --version:

Docker Compose version v2.0.0-beta.6

Output of docker version:

Client:
 Cloud integration: 1.0.17
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.16.4
 Git commit:        f0df350
 Built:             Wed Jun  2 12:00:56 2021
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

For me, the easiest solution was to "downgrade" docker-compose back to 1.28 by running docker-engine gui, go to Settings -> Experimental Features -> Unselect Use Docker Compose v2. For me it was a real badass move to not allow to skip update and then turning this Experimental Feature on by default. This was hard to investigate for some time and took me some frustrating hours.

[ndeloof]

We for sure can fix this but https://github.com/docker/compose-cli/issues/984 would be a better way so that we don't depend on original compose file and variables

[pascallandau-ay]

Just ran into the exact same issue and can absolutely share @Fabi89's frustration. I'm using Docker Desktop for Win and the docker-composer v2 was "silently" activated with a minor version upgrade from 3.4.0 to 3.5.0.

It took me quite some time to a) understand the different env resolution behavior b) figure out, that there is no "parity" between up and down when it comes to the used .env file c) config is not helpful here, because it behaves like up - so it can't be used to understand the issue of down

Curiously, down does NOT fail if no -p option is used - which made it even harder to reproduce the error and understand what's going on.

---

My setup to reproduce the issue:

foo/
- docker-compose.foo.yml
- Dockerfile
- .env

# docker-compose.foo.yml
version: '3.7'

networks:
  foo:
    driver: ${NETWORKS_DRIVER}

services:
  testa:
    build:
      context: ./
      dockerfile: Dockerfile
      args:
        - FOO=${FOO}
    networks:
      foo:
        aliases:
          - ${NETWORK_ALIAS}

Starting the containers without -p:

$ docker-compose -f foo/docker-compose.foo.yml --env-file=./foo/.env up -d
docker/compose-cli#1 [internal] load build definition from Dockerfile
[...]
Network foo_foo  Creating
Network foo_foo  Created
Container foo_testa_1  Creating
Container foo_testa_1  Created
Container foo_testa_1  Starting
Container foo_testa_1  Started

$ docker-compose -f foo/docker-compose.foo.yml --env-file=./foo/.env down
Container foo_testa_1  Stopping
Container foo_testa_1  Stopping
Container foo_testa_1  Stopped
Container foo_testa_1  Removing
Container foo_testa_1  Removed
Network foo_foo  Removing
Network foo_foo  Removed

Starting the containers with -p:

$ docker-compose -p some_project -f foo/docker-compose.foo.yml --env-file=./foo/.env up -d
docker/compose-cli#1 [internal] load build definition from Dockerfile
[...]
Network foo_foo  Creating
Network foo_foo  Created
Container foo_testa_1  Creating
Container foo_testa_1  Created
Container foo_testa_1  Starting
Container foo_testa_1  Started

$ docker-compose -p some_project -f foo/docker-compose.foo.yml --env-file=./foo/.env down
time="2021-07-20T17:01:02+02:00" level=warning msg="The NETWORKS_DRIVER variable is not set. Defaulting to a blank string."
time="2021-07-20T17:01:02+02:00" level=warning msg="The FOO variable is not set. Defaulting to a blank string."
time="2021-07-20T17:01:02+02:00" level=warning msg="The NETWORK_ALIAS variable is not set. Defaulting to a blank string."
services.testa.networks.foo.aliases.0 must be a string

[bigwhoop]

Experiencing the same issue as @pascallandau-ay. The .env file in combination with a --project-name doesn't work on down. Version 2.0.0-beta6 as well.

$ ls -a
.  ..  .env  docker-compose.yml

$ cat .env
IMAGE=schickling/mailcatcher

$ cat docker-compose.yml
version: '3.4'

services:
  test:
    image: ${IMAGE}

$ docker-compose --project-name=test up -d
[+] Running 1/0
 ⠿ Container test_test_1  Running

$ docker-compose --project-name test down
WARN[0000] The IMAGE variable is not set. Defaulting to a blank string.
services.test.image must be a string

[bigwhoop]

To give a brief update for the test case 🔝 with v2.0.0-rc.3. Issue still exists, but error message is slightly different.

$ docker-compose --version
Docker Compose version v2.0.0-rc.3

$ docker-compose --project-name test down
WARN[0000] The "IMAGE" variable is not set. Defaulting to a blank string.
service "test" has neither an image nor a build context specified: invalid compose project

[bigwhoop]

The use case I outlined above now works without a warning.

$ docker --version
Docker version 20.10.10, build b485636

$ docker-compose --version
Docker Compose version v2.1.1

$ docker-compose --project-name=test up -d
[+] Running 2/2
 ⠿ Network test_default   Created
 ⠿ Container test-test-1  Started

$ docker-compose --project-name test down
[+] Running 2/2
 ⠿ Container test-test-1  Removed
 ⠿ Network test_default   Removed

Also no issue when using docker compose instead of docker-compose.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.