SecretsManager secrets by name

[al-dpopowich]

This was recently discussed on gitter, where it was declared a documentation bug. A patch was made to the documentation, but I still believe a useful feature is missing.

Documentation once suggested you could reference a secret by name, e.g.:

secrets:
   APP_CONFIG:
      secretsmanager: ${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/app-config

But as noted in the gitter discussion, this results in a PermissionDenied error because the generated CFN does not include the 6-char randomness that SecretsManager appends to each ARN:

   Secrets:
     - Name: APP_CONFIG
       ValueFrom: !Sub 'arn:${AWS::Partition}:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:myapp/myenv/app-config'

This means we cannot write concise manifests, but will have to write environment overrides for each and every service for each environment to include the 6-char randomness:

environments:
   beta:
      secrets:
         APP_CONFIG: 
            secretesmanager: ${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/app-config-kas892
   staging:
      secrets:
         APP_CONFIG: 
            secretesmanager: ${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/app-config-ss1cw93

Having to touch every service manifest is quite painful and error prone.

I believe copilot needs to fetch the ARN by name and place it in the CFN rather than building it manually, e.g., the equivalent of:

$ aws --output yaml secretsmanager describe-secret --secret-id myapp/myenv/app-config | yq '.ARN'

Also, if it fetched the ARN prior to deployment, as part of building the CFN, it would be a better user experience to find out immediately that the secret doesn't exist (or is not properly tagged!) rather than receiving a cryptic message mid-deployment.

Thanks!

[huanjani]

Hello, @al-dpopowich.

Try taking out the app and env names; you shouldn't need the string of random chars. We updated the docs to reflect that app and env names shouldn't be in there. Ref: https://app.gitter.im/#/room/#aws_copilot-cli:gitter.im/$Ic3SPmfjD_W3c7_Qax-oVaebh9YVY4o1fdV1JTuyhnw https://github.com/aws/copilot-cli/pull/5724

Thanks!

[huanjani]

Leaving this comment open for the errors messaging part! Thanks.

[al-dpopowich]

That my SecretsManager secret uses app and env names is immaterial. When I try to reference a Secret by name, the CFN uses exactly that name, as part the ARN it generates in the CFN, but no such secret exists (because it lacks the randomness), so fails during deployment.

E.g., if in my manifest I have this:

secrets:
   FOO:
      secretsmanager: foo

The following is generated in my CFN:

      ContainerDefinitions:
        - Name: !Ref WorkloadName
          Image: !Ref ContainerImage
          Secrets:
            - Name: FOO
              ValueFrom: !Sub 'arn:${AWS::Partition}:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:foo'

But no such secret exists! The actual ARN is:

arn:aws:secretsmanager:us-east-2:XXXXXXXXXXXX:secret:foo-CMahiK

[mattattui]

I have the same issue as @al-dpopowich and can confirm that a copilot svc package reveals that providing a secretsmanager value (with or without a path) always resolves to a !Sub 'arn… statement which does not resolve to the real secret's ARN. Perhaps there's been a regression since Janice Huang's success on Gitter.

The cloudformation documentation makes clear that an ARN is the only possible way to provide a secrets manager value here (Systems Manager parameter store secrets can be referred to by path, but not Secrets Manager secrets), so I assume that copilot-cli usually performs some cleverness to do a lookup and this isn't functioning for either of us. Either that or the documentation is still wrong.

I've experimented with instead exporting a value from the workload addon, since those get declared as environment variables in the ECS service (in this example as APP_DATABASE_PASSWORD and APP_DATABASE_USER):

Outputs:
  AppDatabaseUser:
    Description: "The username of the database server."
    Value:
      !Join ["", ["{{resolve:secretsmanager:", !Ref DatabaseSecret, ":SecretString:username}}"]]
    Export:
      Name: !Sub ${App}-${Env}-${Name}-AppDatabaseUser
  AppDatabasePassword:
    Description: "The password of the database server."
    Value:
      !Join ["", ["{{resolve:secretsmanager:", !Ref DatabaseSecret, ":SecretString:password}}"]]
    Export:
      Name: !Sub ${App}-${Env}-${Name}-AppDatabasePassword

This works, but obviously it's not ideal, as the value of the secret appears in the task definition.

[Lou1415926]

The original response:

@al-dpopowich I think I might have found out why it (not attaching the random 6 letters) worked for @huanjani but not for you.

In many cases, Secrets Manager can find your secret from part of an ARN rather than the full ARN. However, if your secret's name ends in a hyphen followed by six characters, Secrets Manager might not be able to find the secret from only part of an ARN. Instead, we recommend that you use the complete ARN or the name of the secret. - doc

Your secret name is app-config with the hyphen, so omitting -ss1cw93 didn't work for you.

For context, I personally tested the following:

1. Create a secret called "foo". The secret arn is arn:aws:secretsmanager:us-west-2:...:secret:foo-RANDOM
2. Run aws secretsmanager get-secret-value --secret-id foo. Worked ✅
3. Add the following block to one of my services and deploy it. Worked ✅

secrets:
  FOO:
    secretsmanager: foo

I think if my secret name had been ~foo-bar~ foo-barbar, then I'd need to specify ~foo-bar-RANDOM~ foo-barbar-RANDOM instead of just ~foo-bar~ foo-barbar.

Edit:

@mattattui Maybe it was the same problem with your service! Similar to what I said about, you could refer to the secret using a partial arn - an ARN without the random string - if the secret name doesn't contain a hyphen. Copilot constructs the ARN from what you provide for secrets.[NAME]. If the name you provide is name-<RANDOM>, then the constructed ARN is a full ARN; if you omit -<RANDOM>, then the constructed ARN is a partial ARN.

[mattattui]

Oh good lord 😃 Great sleuthing! I'll give it a go on Monday. If that turns out to be the trick, that caveat surely merits a documentation update.

[mattattui]

Quick followup to @Lou1415926's suggestion. Indeed if I create a secret in a workload addon with the name myapp/myenv/secretwithnohyphens, then I can retrieve it in a workload manifest with

secrets:
  TEST_SECRET:
    secretsmanager: '${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secretwithnohyphens:password::'

I also tried a name with a hyphen but with more than 6 letters: db-credentials and that worked too. It's unfortunate that the random suffix is 6 letters, because I think xxxx-secret is a pretty reasonable name for a secret, and I doubt I'll be alone in discovering this problem 🙂

Finally, I can't simply use 'secretwithnohyphens:password::' as suggested, but that's not surprising given the name of the secret. Thanks again for figuring out the problem and for updating the documentation.

[al-dpopowich]

I can confirm that changing the name of my secretmanager's secret from myapp/myenv/app-config to myapp/myenv/appconfig works and allows me to reference the secret once at the top-level without requiring environment overrides:

secrets:
   APP_CONFIG:
      secretsmanager: ${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/appconfig

fercryinoutload! :zany_face:

[ssyberg]

I'm not totally following what we're supposed to change here, I have this working in one environment but not a newly initialized environment. Is there a summary / changelog that explains what you need to update for this to work now? Was this introduced in a recent CLI update?

[ssyberg]

Is the suggestion that I can just change from this:

secrets:
    SECRET_KEY:
        secretsmanager: "${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-secrets:SECRET_KEY::"

To this?

secrets:
    SECRET_KEY:
        secretsmanager: SECRET_KEY

And copilot will just figure it out? That seems wrong since there's no way for copilot to know my secrets are in something that ends in -secrets

[ssyberg]

I reinterpreted the above and tried this:

secrets:
    SECRET_KEY:
        secretsmanager: "${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}:SECRET_KEY::"

But now even the previously deployable services are broken.

[ssyberg]

Apologies, I had missed the note about adding tags, adding these manually fixed this!

Copilot requires the copilot-application and copilot-environment tags to limit access to this secret.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no response activity. Remove the stale label, add a comment, or this will be closed in 14 days.

[github-actions[bot]]

This issue is closed due to inactivity. Feel free to reopen the issue if you have any further questions!