Environment variable based volume mount source override is broken in v2

[vlaci]

Description

Steps to reproduce the issue: Given the following compose file docker-compose up works as expected

docker-compose.yml

version: "3.9"
services:
  alpine:
    image: alpine
    volumes:
      - type: bind
        source: ${BIND_MOUNT_SOURCE}
        target: /src
    command: ["stat", "/src/docker-compose.yml"]

Output

$ BIND_MOUNT_SOURCE=. docker-compose up
[+] Running 2/2
 ⠿ alpine Pulled
   ⠿ 530afca65e2e Pull complete
[+] Running 1/0
 ⠿ Container test-alpine-1  Recreated
Attaching to test-alpine-1
test-alpine-1  |   File: /src/docker-compose.yml
test-alpine-1  |   Size: 134       	Blocks: 8          IO Block: 4096   regular file
test-alpine-1  | Device: 25h/37d	Inode: 894393      Links: 1
test-alpine-1  | Access: (0644/-rw-r--r--)  Uid: ( 1000/ UNKNOWN)   Gid: (  100/   users)
test-alpine-1  | Access: 2022-07-28 13:19:41.254177496 +0000
test-alpine-1  | Modify: 2022-07-28 13:19:40.111170699 +0000
test-alpine-1  | Change: 2022-07-28 13:19:40.111170699 +0000
test-alpine-1 exited with code 0

If I want to set a default value for the variable, I could use an override:

docker-compose.override.yml

version: "3.9"
services:
  alpine:
    image: alpine
    volumes:
      - type: bind
        source: .
        target: /src

Output

$ docker-compose up
WARNING: The BIND_MOUNT_SOURCE variable is not set. Defaulting to a blank string.
Starting test_alpine_1 ... done
Attaching to test_alpine_1
alpine_1  |   File: /src/docker-compose.yml
alpine_1  |   Size: 192       	Blocks: 8          IO Block: 4096   regular file
alpine_1  | Device: 25h/37d	Inode: 894393      Links: 1
alpine_1  | Access: (0644/-rw-r--r--)  Uid: ( 1000/ UNKNOWN)   Gid: (  100/   users)
alpine_1  | Access: 2022-07-28 13:30:41.935903837 +0000
alpine_1  | Modify: 2022-07-28 13:27:19.264792339 +0000
alpine_1  | Change: 2022-07-28 13:27:19.264792339 +0000
test_alpine_1 exited with code 0

Describe the results you received: Using v2 it no longer works:

$ docker-compose up
WARN[0000] The "BIND_MOUNT_SOURCE" variable is not set. Defaulting to a blank string. 
invalid mount config for type "bind": field Source must not be empty

When no variables are used, or

Describe the results you expected:

I expect the above setup to work as before: the mount specifications should be merged based on the volume target.

Additional information you deem important:

Some rationale for our setup: we use the compose file as a basis for docker stack deployments, and we use .override.yml files to contain defaults for local development (not only envvars, but build specification in addition to image and so on).

Output of docker compose version:

Docker Compose version 2.7.0

Output of docker info:

docker info output

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., 0.0.0+unknown)
  compose: Docker Compose (Docker Inc., 2.7.0)

Server:
 Containers: 56
  Running: 13
  Paused: 0
  Stopped: 43
 Images: 444
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: v1.6.6
 runc version: 
 init version: 
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.53
 Operating System: NixOS 22.11 (Raccoon)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 46.73GiB
 Name: tachi
 ID: TPJ3:WLRE:HH42:KBLP:H4CL:W6L4:BDC4:VEBN:DPTY:557D:DSGY:KX6G
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: onekeysec
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

Additional environment details: The issue doesn't seem to be environment specific

[ulyssessouza]

I'm not sure if I understood the issue, but in the case of using a default value for the variable you could use the default value notation on environment variables (":-" for cases of empty or unset variables). The compose file would look like:

version: "3.9"
services:
  alpine:
    image: alpine
    volumes:
      - type: bind
        source: ${BIND_MOUNT_SOURCE:-.}
        target: /src
    command: ["stat", "/src/docker-compose.yml"]

[vlaci]

The reason we didn't want to go this direction, is tat we wanted to be certain not to use the default value in production. That's why we are using a separate override which is not present when we feed the compose files to docker stack deploy.

[drewyangdev]

If I understand this correctly, I'm experiencing the same issue.

Re-produce

docker-compose.yaml

version: '2.4'
services:
  my_test:
    build:
      context: .
      args:
        - PY_VER
    image: my_test_image:py${PY_VER}-debian
    environment:
      - CONTAINER_USER
      - IMAGING_ROOT_DATA_DIR=/home/${CONTAINER_USER}/inbox
      - IMAGING_PROCESSED_DATA_DIR=/home/${CONTAINER_USER}/outbox
    volumes:
      - ${ROOT_DATA_DIR}:/home/${CONTAINER_USER}/inbox
      - ${PROCESSED_DATA_DIR}:/home/${CONTAINER_USER}/outbox

build.env

PY_VER=3.9

Dockerfile

FROM debian:latest
RUN apt-get update

Build by docker-compose(v1)

# build
set -a
source build.env
docker-compose build

# output
WARNING: The CONTAINER_USER variable is not set. Defaulting to a blank string.
WARNING: The ROOT_DATA_DIR variable is not set. Defaulting to a blank string.
WARNING: The PROCESSED_DATA_DIR variable is not set. Defaulting to a blank string.
Building standard_worker
[+] Building 5.3s (4/6)                                                                             
[+] Building 5.4s (5/6)                                                                             
 => [internal] load build definition from Dockerfile                                           0.0s
 => => transferring dockerfile: 80B                                                            0.0s
 => [internal] load .dockerignore                                                              0.0s
 => => transferring context: 2B                                                                0.0s
 => [internal] load metadata for docker.io/library/debian:latest                               1.5s
 => [auth] library/debian:pull token for registry-1.docker.io                                  0.0s
 => CANCELED [1/2] FROM docker.io/library/debian:latest@sha256:82bab30ed448b8e2509aabe21f40f0  3.7s
 => => resolve docker.io/library/debian:latest@sha256:82bab30ed448b8e2509aabe21f40f0607d905b7  0.0s
 => => sha256:82bab30ed448b8e2509aabe21f40f0607d905b7fd0dec72802627a20274eba5 1.85kB / 1.85kB  0.0s
 => => sha256:3098a8fda8e7bc6bc92c37aaaa9d46fa0dd93992203ca3f53bb84e1d00ffb796 529B / 529B     0.0s
 => => sha256:07d9246c53a664dde2b72c3d531d19dff4205a02bdd0e9612a5500aa51b8630 1.46kB / 1.46kB  0.0s
 => => sha256:001c52e26ad57e3b25b439ee0052f6692e5c0f2d5d982a00a8819ace5e521 18.87MB / 55.00MB  3.7s

Build by docker compose(v2)

# build
set -a
source build.env
docker compose build

# output
WARN[0000] The "ROOT_DATA_DIR" variable is not set. Defaulting to a blank string. 
WARN[0000] The "CONTAINER_USER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "PROCESSED_DATA_DIR" variable is not set. Defaulting to a blank string. 
WARN[0000] The "CONTAINER_USER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "CONTAINER_USER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "CONTAINER_USER" variable is not set. Defaulting to a blank string. 
2 error(s) decoding:

* error decoding 'Volumes[0]': invalid spec: :/home//inbox: empty section between colons
* error decoding 'Volumes[1]': invalid spec: :/home//outbox: empty section between colons

---

Comment

It seems like when docker-compose build parse the docker-compose.yaml file with the env variables that loaded from build.env, it only cares about the yaml sections: build and image. So if the env variable is unset by user and set by default to blank string in the volume section, it doesn't matter for the build. However, for docker compose build, it parses all the sections, even volume section is not necessary for build at this point.

[ndeloof]

docker compose v2 indeed does:

• parse a yaml file into a tree
• interpolate variables
• map tree to internal model based on go structs
• merge with any previously loaded file

doing so, each individual file is validated, and missing variable is reported to user. also, the whole compose file is parsed and validated, without consideration for the command being ran.

Changing this to adopt the same approach as docker compose v1 (parse yaml into a tree, merge trees, then interpolate/use model) is technically feasible but is a significant effort, I don't expect this to happen short terms

[storm1er]

I'm using this to differ ci/cd compose.yml with a local compose.override.yml for images tag:

compose.yml:

services:
  web:
    image: registry.gitlab.com/<group>/<project>/<image>:$TAG
...

compose.override.yml:

services:
  web:
    image: registry.gitlab.com/<group>/<project>/<image>:dev
...

I'm doing this for 2 images in compose, I'm getting this each time locally :(

❯  docker compose pull
WARN[0000] The "TAG" variable is not set. Defaulting to a blank string. 
WARN[0000] The "TAG" variable is not set. Defaulting to a blank string. 
[+] Pulling 23/12
 ✔ worker Pulled                                                     39.8s 
 ✔ database 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled       17.3s 
 ✔ web 8 layers [⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                 39.8s 

Which is kind of annoying as new devs think there's a problem but there's not :/

[ndeloof]

@storm1er you could declare variable with a default value, so you don't even need an override file in development:

services:
  web:
    image: registry.gitlab.com/<group>/<project>/<image>:${TAG:-dev}

[ndeloof]

tested with https://github.com/docker/compose/pull/11207 and confirmed issue is fixed using compose-go/v2