[BUG] Container named volume mount source not getting updated

[g-kartik]

Description

I want postgres database to use separate data volumes for each git branch. I am passing the CURRENT_GIT_BRANCH value through .env file.

Here is the docker compose file

services:
    cvat_db:
        image: postgres:15-alpine
        environment:
            POSTGRES_USER: root
            POSTGRES_DB: cvat
            POSTGRES_HOST_AUTH_METHOD: trust
        volumes:
            - cvat_db:/var/lib/postgresql/data

volumes:
    cvat_db:
        name: cvat_db_{CURRENT_GIT_BRANCH}

When executing docker compose up for the first time with CURRENT_GIT_BRANCH as develop, the container gets correctly mounted with the volume cvat_db_develop.

With a new value for the CURRENT_GIT_BRANCH, say feature, the volume cvat_db_feature gets created successfully, however the container volume mount source still refers to the old cvat_db_develop volume.

The expected behavior is that by changing the name of the named volume:

• A new volume should be created with the given name (happens correctly)
• The source mount path of all the containers referencing that volume should refer to the new volume. (does not happen)

Compose Version

Docker Compose version v2.24.5-desktop.1

Docker Environment

Client:
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5-desktop.1
    Path:     /usr/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     /usr/lib/docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /usr/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /usr/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /usr/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     /usr/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.4.1
    Path:     /usr/lib/docker/cli-plugins/docker-scout

Server:
 Containers: 8
  Running: 8
  Paused: 0
  Stopped: 0
 Images: 22
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 64b8a811b07ba6288238eefc14d898ee0b5b99ba.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.5-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 19.45GiB
 Name: archlinux
 ID: c755af18-ccd5-4262-9be4-a17d19c101b2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: gkarthikraja
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[ndeloof]

Only services indeed get recreated after configuration change has been detected. We could do the same for volumes (while bit more complex to manage regarding lifecycle) but this would mean user data would be deleted, which could be an issue.

[g-kartik]

but this would mean user data would be deleted, which could be an issue.

We can persist the old volume without deleting it and just mount the new volume. Can we do that?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as not stale anymore due to the recent activity.

[g-kartik]

@ndeloof Can we keep the old volumes without deleting and mount the new one?

[ndeloof]

@g-kartik nope, as volume name is unique we have no option to keep the existing one

[g-kartik]

I did not understand. The issue is regarding when volume names changes in the compose file, the docker compose up command should unmount (but not delete) the old volume and mount the new volume. If we again change back to old volume then it should get mounted again and reused. There is no issue of uniqueness here.

[ndeloof]

right, my bad, have been reading too fast. The issue here is that compose compares actual container state with desired state only based on service.foo yaml section, where volume is referenced as cvat_db in the reported example. The fact actual volume name is updated (or any other configuration attribute) isn't detected. A comparable issue is https://github.com/docker/compose/issues/11900

IMHO we miss a global mechanism to detect volume/config/secrets/networks do not run with desired state, and recreate services accordingly