[ECS] [Volumes]: Ability to create config "volume" and mount it into container as file

[taraspos]

Tell us about your request Would be very nice to be able to mount strings (secrets/configurations) defined in the task definition into the container as a file.

Which service(s) is this request for? ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Some apps need to be configured via config file on the file system and in order to put it there, we need to build/store a custom image. This is very uncomfortable, especially in the case when the same container can be reused with just as small configuration difference. Being able to define a config file as a "volume" and mount into the container would solve the issue, actually, k8s can do that.

Additional context This could be used for secrets as well (#8). Kubernetes does it via ConfigMaps

[FernandoMiguel]

Sounds like you need the next version of Docker engine, that will have that from buildkit

[taraspos]

You are talking about the experimental features? That's nice, but I would like to be able to do it during the start of the container, not during the build of the image, or I understood what it does wrong.

[FernandoMiguel]

@Trane9991 oh right 🤭

[mumoshu]

This is affecting the Fargate provider for virtual-kubelet as well.

https://github.com/virtual-kubelet/virtual-kubelet/issues/185#issuecomment-452542691

To maintain feature parity with Kubernetes configmap/secret, an ECS API that updates the "config" object, with the ability to stream config updates to the container volume would be ideal.

[jpalomaki]

Now that ECS Fargate seems to have native support for passing in AWS Secrets Manager secrets via environment variables, one possible workaround, for being able to mount secrets as files into an ECS task (app container):

Run a secrets-holding sidecar container in the ECS task, and pass Base64-encoded Secrets Manager secrets to it via environment variables, and have that sidecar Base64-decode and save those secrets in its file system, in e.g. /etc/secrets.d, root-owned, but readable by application user. Then we could use the volumesFrom (with sourceContainer pointing to sidecar) ECS container definition stanza to mount that directory (with the secrets) to the app container, read-only?

Pros:

• No environment variables (with secrets) are exposed to the main (app) container?
• Secrets Manager access is limited to Task exec IAM role (task IAM role don't need it)?
• No need to do special secrets resolution in the app container image (e.g. CMD or ENTRYPOINT script)
• Does this avoid Secrets manager API throttling, as the secrets are injected in task exec phase and not pulled in at app container startup?

Cons:

• Uses a volume (not tmpfs I presume) to hold secrets, dunno about the security implications of that? Are Fargate volumes EBS-encrypted? How about inter-task/container isolation?
• Some complexity in setting this up in the task definition, though also fairly easily ejectable when we get the native support for this?

Any thoughts on this workaround and, the pros and cons, and in particular, its security implications in the ECS Fargate environment?

[philandstuff]

We have a similar workaround to @jpalomaki, except without the sidecar. It relies on having access to a shell (sh or bash) in the underlying container.

The trick is: you set your file contents as base64-encoded environment variables, then override entrypoint to sh -c, and insert a shell snippet that base64-decodes the envvars into the appropriate files on disk, then starts the binary at the original entrypoint. (We base64-encode them to make it easier to include in the task definition JSON file; in principle, base64 is not required; especially for environment variables populated from Parameter Store.)

Pros:

• no sidecar container
• no race condition (a sidecar runs concurrently with the primary container, so there's a race to write the config file before the primary container reads it)

Cons:

• ugly :disappointed:
• doesn't work with containers that don't have a shell included

Here's an example task definition JSON doing this trick for prometheus. It's an almighty hack, but until a feature like this lands, it's better than nothing.

[magrossm]

@philandstuff I have the same problem with an awx installation on Fargate. I had the same idea as you, since the idea of @jpalomaki seemed a bit too much storage-wise (costs), but it the container doesnt come because it tells me Permission Denied since its in /etc/. I will try it with PArameter Store but still that wouldnt solve the permission issue. any advice?

[matthewcummings]

I like this idea a lot, would be helpful for me with ECS and EKS. I deployed a RabbitMQ image the other day - being able to mount a pre-canned config volume would be convenient.

[jpalomaki]

@philandstuff Looks like the sidecar startup race condition might be avoidable: see dependsOn at https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html

[zhangalex]

I'm trying to put the config/secret file on S3 then read it from the application. The idea comes from Aliyu Container Service, maybe ECS can internally support it.

[jpalomaki]

@abby-fuller Any updates on the progress on this one?

[s-i-l-k-e]

How is this progressing? This is absolutely a must have to avoid having to bake in (minor) config changes to public images.

[cmutzel]

+5

[TapTap21]

Any news on this? Adding side-cars or customized entrypoints to achieve this isn't great.

[Vlaaaaaaad]

An option to mount a file would also help a lot for larger config files which are now limited by the 8Kb limit for SSM Advanced Parameters and 10Kb limit for Secrets Manager. Creating files from multiple parts is... particularly terrible.

[slmg]

We solved this issue with the following process: Push config to Github -> CD push to S3 bucket -> lambda function copy config from S3 to an EFS share -> EFS gets mounted as a volume in containers.

[JohnPreston]

I actually find the side-car option rather convenient, especially using Secret Manager as the source of secrets or SSM for config.

Pros:

1. If your application requires a specific file format for the configuration, your side-car can then insure that the right format is respected. For example if you want to declare variables in PHP, that won't be the same format(or formatting anyway) as if your application has a well known config format (for example, [app.db]).

2. you could define a shared volume between your side car and your application, have the app mount in read-only the file so it cannot modify the values itself. The Execution role gets the grant to fetch the secret but not the apps. If the app is compromised, you can (fix your app first) rotate the secret, re-deploy, and that gets all the things updated.

services:
  app:
    volumes:
      - shared:/path/to/mount:ro
   deploy:
     labels:
       ecs.task.family: app01
    depends_on:
      - secret-fetcher

  secret-fetcher:
    secrets: [secret-01]
   volumes:
     - shared:/path/to/mount:rw
   deploy:
     labels:
       ecs.task.family: app01
       ecs.depends.condition: SUCCESS
     
volumes:
  shared: {}
  
secrets:
  secret-01:
    Name: /path/to/secrets/manager

3. Using the ECS dependency check, you can ensure that the side-car has successfully completed its task prior to starting your app. There is very little point in starting your app if something went wrong in setting the configuration. That could be a con of its own as that means that your side car code has to work ..

4. very versatile, you never have to hardcode anything in the docker image, you can have a volume bind locally to use your local config and not make a change to the definition of your service

Cons. you need to write your own sidecar..

note: using ECS CLI v1 or v2 you would get two separate services etc. To make use of the labels above to group your docker service together in AWS ECS you'd need to use Compose-X to group them in the same task definition and implement the dependency.

[joebowbeer]

Given that EKS can already do this, why not use to EKS?

Meta/Aside: Is it reasonable to expect ECS to achieve feature parity with EKS?

[mreferre]

@joebowbeer that's an interesting philosophical question. I think the answer is no. Kubernetes (and hence EKS) over-indexes on flexibility while ECS over-indexes on ease of use (yes there is still work to do). Also, Kubernetes tries to be a platform itself (overlaying heterogeneous infrastructures) whereas ECS has a more narrow role in the context of a larger set of services that AWS offers (and that ECS leverages as a "peer").I see them evolving in different directions. This doesn't mean that if a feature is interesting/useful that should not be implemented in ECS (and the fact that a feature exists in EKS may not be enough to move the house from ECS to EKS).

[reachlin]

anyone checked out this? ECS with EFS volumes? looks like a better solution? https://www.infoq.com/news/2020/04/aws-ecs-fargate-efs-support/ @Trane9991 fyi.

[tarfeef102]

anyone checked out this? ECS with EFS volumes? looks like a better solution? https://www.infoq.com/news/2020/04/aws-ecs-fargate-efs-support/ @Trane9991 fyi.

Not really "better", because if you, say, use terraform for all your infra, you can't exactly provision an EFS volume with the data you want in it either, so it introduces manual steps into your deployment process.

In the meantime I personally created a docker container that acts as a sidecar and exposes a volume in which it grabs stuff from S3 and slaps there. That at least allows for automated deployments to happen with our stack. But I'd love to see a cleaner, sidecar-less solution

[reachlin]

anyone checked out this? ECS with EFS volumes? looks like a better solution? https://www.infoq.com/news/2020/04/aws-ecs-fargate-efs-support/ @Trane9991 fyi.

Not really "better", because if you, say, use terraform for all your infra, you can't exactly provision an EFS volume with the data you want in it either, so it introduces manual steps into your deployment process.

In the meantime I personally created a docker container that acts as a sidecar and exposes a volume in which it grabs stuff from S3 and slaps there. That at least allows for automated deployments to happen with our stack. But I'd love to see a cleaner, sidecar-less solution

i won't use terraform to deploy my app. terraform is good at infra. provision, but sucks at app deployment and configuration. just my two cents.

[tarfeef102]

i won't use terraform to deploy my app. terraform is good at infra. provision, but sucks at app deployment and configuration. just my two cents.

Not really too on-topic so I won't drag this out, but there are many situations/places where ECS clusters, services, tasks, and even containers are defined by terraform, and having features like discussed here would help. The point of tools is to give the users features they find useful, not leave them by the wayside because they prefer doing things another way, especially when "other ways" are not exactly a 10 minute job to switch to.

[cageyv]

Okay, we need new types of Volume in Task Definition. Secret Type and Parameter Type Hmm, probably we need new data types are needed in the Parameter Store and Secret Manager. For example "file", "base64encoded", "base64 + gziped" (as is EC2 user-data) And all this is necessary because this method ("whole file") of transferring configurations is typical for the yaml format, which is more actively used in k8s than in ECS. (in ECS it seems to me that there is more JSON) Or perhaps some applications violate the 12-factor principle a little and do not want to receive values from environment variables. In some cases, we can get out of the situation in another way. For example in the case of Filebeat from Elastic I do it with overriding CMD + existing ENV.

It seems to me that a unified solution with a side-car that takes data from S3 / ParameterStore / Secret Manager would also be a good option. Perhaps something similar to awsfirelens for sending logs. "Custom config files"

At the moment there is already a project "AWS Secret Sidecar Injector" And one of its parts is "aws-secrets-manager-secret-sidecar" Perhaps all that remains is to turn the base64encoded values of the secret key into a file, and the file name will be the key itself.

Also, in general, storing secret values in mounted volumes may not be the best idea, since they can be connected again via bind-volumes, but without any control or restrictions. I still think that the lack of this function is not the reason for the transition from ECS to EKS

[sj26]

Yeah I'd like two different versions of this.

The first version is that I want to use unmodified docker images from upstreams which presume the mounting of configuration files without any secrets. For this, I'd be perfectly happy to mount a read-only volume which is copied from an s3 object or prefix. Something like:

{
  "volumes": [
    // unsure of syntax, options include:
    // download every object under this prefix into a volume
    {"name": "config", "s3": {"uri":"s3://config.example.com/redis/"}}
    {"name": "config", "s3": {"bucket":"config.example.com", "prefix":"/redis/"}}
    // or individual objects
    {"name": "config", "s3": {"uri":"s3://config.example.com/redis.conf"}}
    {"name": "config", "s3": {"bucket":"config.example.com", "key":"/redis/redis.conf"}}
    // maybe many
    {"name": "config", "s3": {"bucket":"config.example.com", "keys":["/redis/redis.conf", "/redis/cert.pem"]}}
    // maybe archives, a la Dockerfile ADD and layer/volume exports
    {"name": "config", "s3": {"uri":"s3://config.example.com/redis.tar.gz"}}
  ],
  "containerDefinitions": [
    {
      ...,
      "mountPoints": [
        {"sourceVolume": "config", "containerPath": "/etc/redis/"}
      ]
    }
  ]
}

This would be a fine stop-gap for configuration files containing low-risk secrets combined with s3 encryption, too.

It would also be great to be able to mount a secret into the filesystem. Something like:

{
  "containerDefinitions": [
    {
      ...,
      "mountPoints": [
        {"sourceSecret": "arn:aws:ssm:...:/parameter/path/to/ssh/id_rsa", "containerPath": "/app/.ssh/id_rsa"}
      ]
    }
  ]
}

[JohnPreston]

Hello all. Had not seen updates on this before this morning and it is interesting to see a side car project for EKS. So coincidentally, I had created my own version of that for the purpose of doing exactly all of the above for work projects and just got some first images out on ECR.

So, I decided to create my own very simple version of it. It is inspired by the CFN ConfigSet.files in syntax and capability.

The idea is that you can use a SSM parameter or S3 file or whatever so long as the input complies with the input schema in which you can represent the configuration.

I need to add some examples in docker-compose and with compose-x. Using the ECS success condition, the sidecar starts first, does whatever it has to do, on success returns 0 and exits. (dependency in compose-x and in AWS Docs)

From the examples above I will work to add a few features (zip and tar) and will soon work on allowing to render a final file via Jinja.

For now tested with some basic import tasks

files:
  /tmp/test.txt:
    content: >-
      test from a yaml raw content
    owner: john
    group: root
    mode: 600
  /tmp/aws.template:
    source:
      S3:
        BucketName: ${BUCKET_NAME:-sacrificial-lamb}
        Key: aws.yml

  /tmp/ssm.txt:
    source:
      Ssm:
        ParameterName: /cicd/shared/kms/arn
    commands:
      post:
        - file /tmp/ssm.txt

  /tmp/secret.txt:
    source:
      Secret:
        SecretId: GHToken

Not as good as native integration but might as well give it a shot and so if anyone has some feedback please let me know!

[ltardivo]

I actually find the side-car option rather convenient, especially using Secret Manager as the source of secrets or SSM for config.

Pros:

1. If your application requires a specific file format for the configuration, your side-car can then insure that the right format is
   respected. For example if you want to declare variables in PHP, that won't be the same format(or formatting anyway) as if your application has a well known config format (for example, `[app.db]`).

2. you could define a shared volume between your side car and your application, have the app mount in read-only the file so it cannot modify the values itself. The Execution role gets the grant to fetch the secret but not the apps. If the app is compromised, you can (fix your app first) rotate the secret, re-deploy, and that gets all the things updated.

services:
  app:
    volumes:
      - shared:/path/to/mount:ro
   deploy:
     labels:
       ecs.task.family: app01
    depends_on:
      - secret-fetcher

  secret-fetcher:
    secrets: [secret-01]
   volumes:
     - shared:/path/to/mount:rw
   deploy:
     labels:
       ecs.task.family: app01
       ecs.depends.condition: SUCCESS
     
volumes:
  shared: {}
  
secrets:
  secret-01:
    Name: /path/to/secrets/manager

1. Using the ECS dependency check, you can ensure that the side-car has successfully completed its task prior to starting your app. There is very little point in starting your app if something went wrong in setting the configuration.
   That could be a con of its own as that means that your side car code has to work ..

2. very versatile, you never have to hardcode anything in the docker image, you can have a volume `bind` locally to use your local config and not make a change to the definition of your service

Cons. you need to write your own sidecar..

note: using ECS CLI v1 or v2 you would get two separate services etc. To make use of the labels above to group your docker service together in AWS ECS you'd need to use Compose-X to group them in the same task definition and implement the dependency.

 secret-fetcher:

This option is very convenient. I've been trying to use this idea, and I'm wondering what image could be used for the secret-fetcher service (I guess a lightweight one).

In my case, I am looking for a way to copy the postgres database initialization script to the path /docker-entrypoint-initdb.d/db_schema.sql inside the container. The problem I am getting is that apparently the full path of the target that I define is not being used to rewrite the default path /run/ secrets (the container log is "open /run/secrets/docker-entrypoint-initdb.d /db_schema.sql: no such file or directory "). I wonder if you, or someone can share a complete example of use of these side-cars. I will be infinitely grateful for the help.

[tarfeef102]

-Snip-

This option is very convenient. I've been trying to use this idea, and I'm wondering what image could be used for the secret-fetcher service (I guess a lightweight one). In my case, I am looking for a way to copy the postgres database initialization script to the path /docker-entrypoint-initdb.d/db_schema.sql inside the container. The problem I am getting is that apparently the full path of the target that I define is not being used to rewrite the default path /run/ secrets (the container log is "open /run/secrets/docker-entrypoint-initdb.d /db_schema.sql: no such file or directory "). I wonder if you, or someone can share a complete example of use of these side-cars. I will be infinitely grateful for the help.

https://github.com/tarfeef101/aws_s3_sidecar is how i've worked around this

[JohnPreston]

I actually find the side-car option rather convenient, especially using Secret Manager as the source of secrets or SSM for config. Pros:

1. If your application requires a specific file format for the configuration, your side-car can then insure that the right format is
   respected. For example if you want to declare variables in PHP, that won't be the same format(or formatting anyway) as if your application has a well known config format (for example, `[app.db]`).

2. you could define a shared volume between your side car and your application, have the app mount in read-only the file so it cannot modify the values itself. The Execution role gets the grant to fetch the secret but not the apps. If the app is compromised, you can (fix your app first) rotate the secret, re-deploy, and that gets all the things updated.

services:
  app:
    volumes:
      - shared:/path/to/mount:ro
   deploy:
     labels:
       ecs.task.family: app01
    depends_on:
      - secret-fetcher

  secret-fetcher:
    secrets: [secret-01]
   volumes:
     - shared:/path/to/mount:rw
   deploy:
     labels:
       ecs.task.family: app01
       ecs.depends.condition: SUCCESS
     
volumes:
  shared: {}
  
secrets:
  secret-01:
    Name: /path/to/secrets/manager

1. Using the ECS dependency check, you can ensure that the side-car has successfully completed its task prior to starting your app. There is very little point in starting your app if something went wrong in setting the configuration.
   That could be a con of its own as that means that your side car code has to work ..

2. very versatile, you never have to hardcode anything in the docker image, you can have a volume `bind` locally to use your local config and not make a change to the definition of your service

Cons. you need to write your own sidecar.. note: using ECS CLI v1 or v2 you would get two separate services etc. To make use of the labels above to group your docker service together in AWS ECS you'd need to use Compose-X to group them in the same task definition and implement the dependency.

 secret-fetcher:

This option is very convenient. I've been trying to use this idea, and I'm wondering what image could be used for the secret-fetcher service (I guess a lightweight one).

In my case, I am looking for a way to copy the postgres database initialization script to the path /docker-entrypoint-initdb.d/db_schema.sql inside the container. The problem I am getting is that apparently the full path of the target that I define is not being used to rewrite the default path /run/ secrets (the container log is "open /run/secrets/docker-entrypoint-initdb.d /db_schema.sql: no such file or directory "). I wonder if you, or someone can share a complete example of use of these side-cars. I will be infinitely grateful for the help.

I created this which manages multi sources. Use it in prod regularly.

Got multiple examples on https://labs.compose-x.io

[dw]

The task definition snippet below is using the aws-cli official Docker image to run aws s3 sync ... at startup. It has the advantage of only relying on well known public images from a reputable source.

{
  "containerDefinitions": [
    {
      "command": ["s3","sync", "s3://xxx-12312313212312-xxx-xxx-offload-prod-config/", "/conf"],
      "mountPoints": [
        {
          "containerPath": "/conf",
          "sourceVolume": "conf"
        }
      ],
      "image": "amazon/aws-cli:2.4.6",
      "essential": false,
      "name": "fetch-config-at-startup"
    },
    {
      "dependsOn": [
        {
          "containerName": "fetch-config-at-startup",
          "condition": "SUCCESS"
        }
      ],
      "command": ["-c", "/conf/vector.toml"],
      "mountPoints": [
        {
          "readOnly": null,
          "containerPath": "/conf",
          "sourceVolume": "conf"
        }
      ],
      "image": "timberio/vector",
      "essential": true,
      "name": "vector"
    }
  ],
  "volumes": [
    {
      "name": "conf",
    }
  ]
}