Feature request for ECS: allow to reference CloudFormation resources from services

[alek-sys]

Hey team, thanks for great work on docker compose up support in AWS ECS, I think that's a really cool feature and it helps to get your app up and running in AWS in seconds. However I noticed one limitation which require hacking generated CloudFormation template which I would like to avoid. Consider the following example (it's not a full working example, but demonstrates the idea):

services:
  server:
    image: my-server-app
    environment:
      POSTGRES_PASSWORD: "???"

x-aws-cloudformation:
  Resources:
    DBAdminPassword:
      Type: AWS::SecretsManager::Secret
      Properties:
        Description: RDS Password
        GenerateSecretString:
          SecretStringTemplate: '{"username": "user"}'
          GenerateStringKey: password
          PasswordLength: 16
          ExcludeCharacters: "\"@/\\"
    DBInstance:
      Type: AWS::RDS::DBInstance
      Properties:
        AllocatedStorage: 50
        DBInstanceClass: db.t3.small
        Engine: postgres
        DBName: test
        MasterUsername:
          Fn::Sub: "{{resolve:secretsmanager:$${DBAdminPassword}::username}}"
        MasterUserPassword:
          Fn::Sub: "{{resolve:secretsmanager:$${DBAdminPassword}::password}}"
   	...

Here I'm creating a custom RDS instance using CloudFormation, and also configuring the DB instance to use automatically generated AWS secret as a password. CloudFormation resources can reference each other using template functions, e.g. Fn::Sub above. Most template functions also have a shortcut form, so MasterUsername: !Sub "{{resolve:secretsmanager:$${DBAdminPassword}::username}}" is basically the same as above.

The problem is though: how do I reference RDS admin password resource from my Compose service? I tried a couple of ideas.

1. Use CloudFormation template function as a value of an environment variable, i.e.

services:
  server:
    image: my-server-app
    environment:
      POSTGRES_PASSWORD: 
         Fn::Sub: "{{resolve:secretsmanager:$${DBAdminPassword}::password}}"

With that, docker compose up fails to produce CloudFormation template at all, exit code 15. I guess because object values are not supported for env vars.

Using shortcut references doesn't work either:

services:
  server:
    image: my-server-app
    environment:
      POSTGRES_PASSWORD: !Sub "{{resolve:secretsmanager:$${DBAdminPassword}::password}}"

docker compose command succeeds, but in the generated CloudFormation template the !Sub part is missing so it's not a reference anymore.

2. Same goes for secret references under secrets:

secrets:
  postgres:
    external: true
    name: !Ref DBAdminPassword
services:
  server:
    image: my-server-app
    secrets:
      - postgres

With that, !Ref is dropped, and

secrets:
  postgres:
    external: true
    name: 
      Ref: DBAdminPassword
services:
  server:
    image: my-server-app
    secrets:
      - postgres

fails.

So to get this working I have to use yq to post-process generated CloudFormation template (yikes!). What are your thoughts on that, do you think supporting references to CloudFormation resources from Compose service / secrets is possible at all?

[misaon]

exactly same problem, temporary solution for me is define .!Sub:

server:
  environment:
    DATABASE_URL: .!Sub "mysql://${DB_USER}:${DB_PASSWORD}@$${RDSDatabaseInstance.Endpoint.Address}/${DB_NAME}"

and process sed -i 's/.!Sub/!Sub/g' in post-script.

[alek-sys]

Thanks @misaon, I ended up with similar solution - post-process templates with yq to set YAML values I need. And it kind of works, but still feels hacky. Would be nice to hear from Compose team what they think.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[6XGate]

This feature would be helpful for better tuning logging within the compose file using AWS::Logs::LogGroup resources under x-aws-cloudformation.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.