Allow multiple volumes with managed EFS for a single service

[efekarakus]

We should enable more than one storage.volumes with managed EFS for a single service.

Ask

Original ask: https://github.com/aws/copilot-cli/issues/1559#issuecomment-942496804

Today users are limited to a single EFS access point per service:

storage:
  volumes:
    frontend:
      efs:
        uid: 123
        gid: 123
      path: /var/data

If they try to use the same EFS filesystem for multiple volumes like this:

storage:
  volumes:
    v1:
      efs:
        uid: 123
        gid: 123
      path: /etc/mount1
    v2:
      efs:
        uid: 456
        gid: 456
      path: /etc/mount2

We return the following error:

validate "storage": cannot specify more than one managed volume per service

Programming model

1. In order to support multiple access points for a single service we need to transform the following resource into a list: https://github.com/aws/copilot-cli/blob/2cc2403b5f69e63c4f4d61f2b8065cc45b2097b9/internal/pkg/template/templates/workloads/partials/cf/efs-access-point.yml#L2-L3
2. We need to assign a new RootDirectory.Path for the additional access points. Today, we hard-code it to the service name like "frontend". The proposal is to do <svcName>-<volumeName>. https://github.com/aws/copilot-cli/blob/2cc2403b5f69e63c4f4d61f2b8065cc45b2097b9/internal/pkg/template/templates/workloads/partials/cf/efs-access-point.yml#L13-L14
3. Update the IAM policy in the TaskRole to grant access to multiple access points: https://github.com/aws/copilot-cli/blob/2cc2403b5f69e63c4f4d61f2b8065cc45b2097b9/internal/pkg/template/templates/workloads/partials/cf/taskrole.yml#L72-L74

[FraserThompson]

Just in case this helps prioritize this... Multiple access points is essential for running CMSs like Drupal (probably others) which save user files in multiple locations. For example:

Public filesystem in /opt/drupal/web/sites/default/files Private fileystem /opt/drupal/private

A single access point couldn't meet this use case. It's possible with externally managed filesystems, but having all this managed by Copilot would be much more convenient.

[efekarakus]

Hi @FraserThompson ! out of curiosity wouldn't this scenario be satisfied with:

storage:
  volumes:
    frontend:
      efs: true
      path: /opt/drupal

Or am I misunderstanding a step?

[FraserThompson]

Hi @efekarakus ! This would work to persist files, yes, but as far as I can tell it would go against best practices.

The other contents of /opt/drupal come from the Docker image and should only exist in ephemeral storage because they're not stateful. The only stateful data is contained in those two directories.

Ideally you could put all files you want persisted into one directory and make that an EFS volume, but Drupal doesn't offer full control over where these files can be.

Also there are other factors, like we might want to share just the public files with an NGINX sidecar so it can serve static assets. We might have another sidecar running Solr for search, and this will also need to persist the search index somewhere. I don't think it would make sense for all of these to persist their data to the same place.

For reference the AWS example for Drupal on ECS uses one filesystem with multiple access points for each directory they want to persist. Unless I'm misunderstanding something (which is entirely possible) I don't think we can do this right now with Copilot.

[hardeepjethwani]

We should enable more than one storage.volumes with managed EFS for a single service.

Ask

Original ask: #1559 (comment)

Today users are limited to a single EFS access point per service:

storage:
  volumes:
    frontend:
      efs:
        uid: 123
        gid: 123
      path: /var/data

If they try to use the same EFS filesystem for multiple volumes like this:

storage:
  volumes:
    v1:
      efs:
        uid: 123
        gid: 123
      path: /etc/mount1
    v2:
      efs:
        uid: 456
        gid: 456
      path: /etc/mount2

We return the following error:

validate "storage": cannot specify more than one managed volume per service

Programming model

1. In order to support multiple access points for a single service we need to transform the following resource into a list: https://github.com/aws/copilot-cli/blob/2cc2403b5f69e63c4f4d61f2b8065cc45b2097b9/internal/pkg/template/templates/workloads/partials/cf/efs-access-point.yml#L2-L3
2. We need to assign a new RootDirectory.Path for the additional access points. Today, we hard-code it to the service name like "frontend". The proposal is to do <svcName>-<volumeName>. https://github.com/aws/copilot-cli/blob/2cc2403b5f69e63c4f4d61f2b8065cc45b2097b9/internal/pkg/template/templates/workloads/partials/cf/efs-access-point.yml#L13-L14
3. Update the IAM policy in the TaskRole to grant access to multiple access points: https://github.com/aws/copilot-cli/blob/2cc2403b5f69e63c4f4d61f2b8065cc45b2097b9/internal/pkg/template/templates/workloads/partials/cf/taskrole.yml#L72-L74

Hi @efekarakus , Do we have any update on this issue resolution?