Using scale number to adjust mounted volumes?

[Sushisource]

I have a situation where I need to create many instances of a docker container, and they should have mounts that enable them to persist their data to different locations, so they aren't all running into each other. The containers are all worker agents for some non-docker-controlled server.

Essentially I need to be able to do something like this:

services:
    agent:
        volumes:
            - /mnt/dat/agent_${DOCKER_SCALE_NUM}:/data/agent

I found a few references when googling around to something like this, but all of them seemed to have a resolution along the lines of "you don't really want to do this" or "here's some other way to solve your problem that doesn't involve doing this".

I doesn't need to be a sequential number or anything like that - I just need some way to get unique mount points for each one.

Is there an existing way to do this? If not, is anything planned?

Thanks!

[manniche]

I have the same use-case, in which I would like to have a variable number of elasticsearch containers, and have each of them write to their own volume, have their own distinct internal name, expose on distinct ports, etc. E.g. a docker-compose definition such as

  es:
    build: ./es/
    restart: unless-stopped
    environment:
      - node.name=es${DOCKER_SCALE_NUM}
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - es_${DOCKER_SCALE_NUM}:/usr/share/elasticsearch/data
    ports:
      - ${DOCKER_SCALE_NUM+CONSTANT}:9200
    networks:
      - elastic

which when running `docker-compose up --scale , I would expect to get 5 containers, 5 volumes with distinct names, 5 different expose ports (starting from some $CONSTANT, e.g. 9199 to make the first container hit the expected 9200 expose port or similar) and a single network they all connect to.

[ruimoliveira]

no response yet and no workaround either?

[let4be]

Seems like docker is not a suitable solution for such scenario - it heavily smells with orchestration, so it's a good fit for kubernetes or nomad

Would be nice to hear from devs if there are any plans and what is the recommended course of action when you need such kind of functionality...

[KingPin]

its a shame that this never got any answer, this seems like a needed part of --scale.

[Soneji]

I would also like to see this, is there any update/info from the docker team?

Sorry for tagging you but it seems that this may have been lost amongst the issues... @shin- @aanand @ulyssessouza @aiordache ?

[lcsondes]

Seems like docker is not a suitable solution for such scenario - it heavily smells with orchestration, so it's a good fit for kubernetes or nomad

There's a pretty wide gap between copying a container a few times and an entire k8s cluster. This would mainly help with cleaning up copypaste and YAML anchors/references - you can almost do this already if not for the missing features proposed above.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Soneji]

bump

[stale[bot]]

This issue has been automatically marked as not stale anymore due to the recent activity.

[dubrsl]

Also got this problem. The solution proposed in this ticket helps to solve.

[apacha]

@glours any thoughts on this? It would be a really nice feature and so far no official response, despite getting around 30 upvotes here.

[buptsb]

bump

[ccmetz]

bump, also have this same issue

[zxcv551133]

bump

[zxcv551133]

No update?

[IainMacdonald]

I have this problem too! The software I use scales the worker containers. When I run my models on those worker nodes it creates a lot of data (thats just a function of the tools installed on the container). I'd like to mount the volume that has all this data on a separate drive but each worker needs a unique mount point... hence the solution above is what I'm looking for!

[ndeloof]

it's not possible we expose container number (used to name replicas) as a variable as this one is computed as we compare existing vs expected resource, AFTER parsing the compose file and processing variables.

To expose a distinct volume per container, you could use anonymous volumes (volume declared in your Dockerfile, not compose.yaml) that engine will create and manage under the hood.

[jamshid]

I'm sure you know but anonymous volume can also be declared in compose.yml, not just a Dockerfile VOLUME.

IMO this is a big missing feature of docker compose. There are many situations where you want a scaled service to each get their own "subdirectory" in a shared volume or to get independent volumes.

services:
  mydb:
    volumes:
      - /var/lib/dbdata

[kant777]

Have the same problem! Any update on this?