support `secrets_file` in the manifest

[KollaAdithya]

conversation from gitter https://app.gitter.im/#/room/#aws_copilot-cli:gitter.im/$iWOLIz9DdgbThFO8oe4-x26Tk1IwT1rHxxoUxwAz9_c

Current Scenario

Say I have two services serviceA and serviceB

.
├── copilot                        (application directory)
│   ├── .workspace                 (workspace summary)
│   ├── serviceA
│   │   └── manifest.yml           (serviceA manifest)
│   ├── serviceB
│   │   └── manifest.yml           (serviceB manifest)

In order to specify secrets for serviceA and serviceB. In ServiceA manifest

secrets:
  GITHUB_WEBHOOK_SECRET: GH_WEBHOOK_SECRET
  DB:
    secretsmanager: 'demo/test/mysql'
  DB_PASSWORD:
    secretsmanager: 'demo/test/mysql:password::'
 
sidecars:
  nginx:
      GITHUB_WEBHOOK_SECRET: GH_WEBHOOK_SECRET
      DB:
        secretsmanager: 'demo/test/mysql'
      DB_PASSWORD:
        secretsmanager: 'demo/test/mysql:password::'
  nginx2:
      GITHUB_WEBHOOK_SECRET: GH_WEBHOOK_SECRET
      DB:
        secretsmanager: 'demo/test/mysql'
      DB_PASSWORD:
        secretsmanager: 'demo/test/mysql:password::'   

In ServiceB manifest

secrets:
  GITHUB_WEBHOOK_SECRET: GH_WEBHOOK_SECRET
  DB:
    secretsmanager: 'demo/test/mysql'
  DB_PASSWORD:
    secretsmanager: 'demo/test/mysql:password::'
 
sidecars:
  nginx:
      GITHUB_WEBHOOK_SECRET: GH_WEBHOOK_SECRET
      DB:
        secretsmanager: 'demo/test/mysql'
      DB_PASSWORD:
        secretsmanager: 'demo/test/mysql:password::'
  nginx2:
      GITHUB_WEBHOOK_SECRET: GH_WEBHOOK_SECRET
      DB:
        secretsmanager: 'demo/test/mysql'
      DB_PASSWORD:
        secretsmanager: 'demo/test/mysql:password::'   

By looking at the manifest files, This feels redundant to specify same secrets across multiple services.

ASK

Instead we can have specify secrets_file in the manifests across services where secrets_file is path to a file from the root of your workspace. (This is similar to env_file in the manifest)

.
├── copilot                        (application directory)
│   ├── .workspace                 (workspace summary)
│   ├── serviceA
│   │   └── manifest.yml           (serviceA manifest)
│   ├── serviceB
│   │   └── manifest.yml           (serviceB manifest)
│   ├── secretsdir
│   │   └── secrets.yml

Where secrets.yml contains all the secrets.

GITHUB_WEBHOOK_SECRET: GH_WEBHOOK_SECRET
DB:
  secretsmanager: 'demo/test/mysql'
DB_PASSWORD:
  secretsmanager: 'demo/test/mysql:password::'   

Copilot should be able to parse this secrets.yml file.

In manifest for serviceA will have

secrets_file: ./secretsdir/secrets.yml

In manifest for serviceB wil have

secrets_file: ./secretsdir/secrets.yml

[KollaAdithya]

A Workaround shared by @interu using yaml Node Anchor and Aliases incase of specifying secrets in a single service. https://github.com/aws/copilot-cli/issues/5203#issuecomment-1683637218

[huanjani]

Related: #3778.

[vskov147]

+1 for this feature request, currently it's a lot of copy/pasta-spaghetti code to update every time we add a new env var that's shared across our many separate service manifests

and unfortunately, the above work-around doesn't help due to the very fact that we have many separate service manifests :(

[sebastianovide]

or a least to let manifest include other files (similarly to the include in docker compose)

[ssyberg]

👍🏼 to this as well, we are having to store the whole .env in secrets manager and do some smelly script work to expose each in the dockerfile

[jlucas91]

Related request: https://github.com/aws/copilot-cli/issues/2699

[ryanc-scalegrowth]

+1 to this issue. Would make implementing new secrets across multiple services much less tedious.