Support Docker Volume Configuration Via Compose

[PettitWesley]

Summary

Support Docker Volume Configuration via the volumes key in Compose.

Description

With #587, support was added for Docker Volume Configuration with ECS Params. Compose provides the fields driver, driver_opts, and labels for docker volumes- the ECS CLI could support reading these fields from Compose. The only difficulty is that in ECS, docker volumes have to be either Task Scoped or Shared Scope- no similar concept exists in Docker Compose. The default for ECS is Task Scoped, however, in Docker Compose, volumes behave more like ECS Shared Scope volumes.

Please +1 this issue if you want to see Docker Compose support for Docker Volume Configuration. Please comment if you have an opinion on Shared Scope vs Task Scoped volumes

[diablodale]

Usually shared scope. And in the following example, myvolume is always shared scope And by specifying the driver, this can cascade to the task definition as it is needed for task placement according to https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DockerVolumeConfiguration.html

version: '2'
volumes:
  myvolume:
    external: true
    driver: 'rexray/ebs:0.11.4'

[diablodale]

I intend to code a fix for a specific use case on this issue. If anyone has suggestions on the below, please comment further on this issue so I can refine the work before starting in January 2020.

I'll add support that will enable ecs-cli v1 to...

P1 functionality - must have

1. docker-compose.yml like the below
2. No matching docker_volumes: section in ecs-params.yml is needed
3. If there is a matching docker_volumes: section for the same volume name in ecs-params.yml it will issue a fatal error and stop

P2 functionality - nice to have, but will release the code without it

1. Should support standard docker-compose merging so to allow merging of volumes: across multiple docker-compose files.

docker-compose.yml

version: '2'
volumes:
  mydata151:                     <--- ecs-params `name`
    external: true               <--- ecs-params `autoprovision: false` and `scope: shared`
    driver: 'rexray/ebs:0.11.4'  <--- ecs-params `driver`
  mys3bucket416:                 <--- ecs-params `name`
    external: true               <--- ecs-params `autoprovision: false` and `scope: shared`
    driver: 'rexray/s3fs:0.11.4' <--- ecs-params `driver`

services:
  webserver:
    image: alpine
    volumes:
      - mydata151:/mnt/mydata
      - mys3bucket416:/mnt/mybucket
  ...

The above will be all that is needed. Just for clarity, the fix will be internally transforming the above into what would previously been required within ecs-params.yml ...

docker_volumes:
  - name: mydata151
    scope: shared
    autoprovision: false
    driver: 'rexray/ebs:0.11.4'
  - name: mys3bucket416
    scope: shared
    autoprovision: false
    driver: 'rexray/s3fs:0.11.4'

[diablodale]

I've started work

[PettitWesley]

👍

[diablodale]

I have a branch that is passing both make test and read-world testing on my live ECS cluster. Anyone want to try/test it before I submit a PR? https://github.com/diablodale/amazon-ecs-cli/tree/fix607-volumes

That branch is based on ecs-cli v1.18.0 and includes

• fix/test cases for this issue
• Makefile updates to enable clearer usage of dep
• Makefile to use dep inside docker

There is one ecs-cli dependency github.com/docker/cli that points to my fork of that project and at the specific commit https://github.com/diablodale/cli/commit/1d4c6d8e492c0598c432314be76e5f59d4c60386. If preferred, the commit after that one on the same branch includes the test cases that pass that project's go test. I've update the Gopkg.toml source url to point to this specific commit until that fix is incorporated into the upstream via a forthcoming PR for https://github.com/docker/cli/issues/2272. I used a specific commit instead of my branch so it is not possible for me to slip other changes into the build.

A real-world docker-compose.yml like the following worked successfully on a live ECS cluster. No ecs-params.yml. Naturally, you have to have the two rexray volume plugins installed + adjust the below yaml for your EBS volume and S3 bucket.

version: '3.4'
volumes:
  my-ebs-volume:
    external: true
    driver: 'rexray/ebs:0.11.4'
  mytestvol2:
  my-s3-bucket:
    driver: 'rexray/s3fs:0.11.4'
    external: true

services:
  webservice:
    image: alpine:3.10
    network_mode: bridge
    volumes:
      - my-ebs-volume:/mnt/one
      - mytestvol2:/mnt/two
      - my-s3-bucket:/mnt/three
    command: tail -f /dev/null

[PettitWesley]

@diablodale Thanks for working on this!

I recommend that you just submit a pull request so that we can start code review.

We won't be able to merge the pull request until your Docker CLI pull request is merged; I'll talk to the rest of the team but I do not think we will want to depend on your fork.