Support labels in AWS SecretsManager backend

[dantonyuk]

Implements versioning through labels in AWS SecretsManager backend.

To define a default label, set spring.cloud.config.server.aws-secretsmanager.default-label. By default it is the active version (AWSCURRENT) is going to be used.

E.g.:

spring:
  profiles:
    active: aws-secretsmanager
  cloud:
    config:
      server:
        aws-secretsmanager:
          region: us-east-1
          default-label: 1.0.0

Note that labelled secrets should have staged labels set, for example:

$ aws secretsmanager create-secret \
      --name /secret/test/ \
      --secret-string '{"version":"1"}'
{
    "ARN": "arn:aws:secretsmanager:us-east-1:012345678990:secret:/secret/version/-a1b2c3",
    "Name": "/secret/version/",
    "VersionId": "cd291674-de2f-41de-8f3b-37dbf4880d69"
}

$ aws secretsmanager update-secret-version-stage \
      --secret-id /secret/test/ \
      --version-stage 1.0.0 \
      --move-to-version-id cd291674-de2f-41de-8f3b-37dbf4880d69

{
    "ARN": "arn:aws:secretsmanager:us-east-1:012345678990:secret:/secret/version/-a1b2c3",
    "Name": "/secret/version/",
}

When a request with 1.0.0 as a label comes, the secrets having staging label 1.0.0 will be picked up.

Relates to #2103.

[codecov[bot]]

Codecov Report

Merging #2104 (96df94a) into main (7c117c0) will increase coverage by 0.02%. The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #2104      +/-   ##
============================================
+ Coverage     77.98%   78.00%   +0.02%     
- Complexity     1500     1503       +3     
============================================
  Files           187      187              
  Lines          5477     5483       +6     
  Branches        710      711       +1     
============================================
+ Hits           4271     4277       +6     
  Misses          937      937              
  Partials        269      269              

Impacted Files	Coverage Δ
...onment/AwsSecretsManagerEnvironmentProperties.java	69.23% <100.00%> (+4.01%)	:arrow_up:
...onment/AwsSecretsManagerEnvironmentRepository.java	93.10% <100.00%> (+0.37%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7c117c0...96df94a. Read the comment docs.

[MatejNedic]

Hey @dantonyuk , I think we should apply different approach to this problem. We should use Secrets Manager Secret staging labels that are tied to Secret. Check versions. After that we can use GetSecretValueRequest versionStage attribute to get it. For me this approach looks more solid and gives us more control. We can then just use for example:

     GetSecretValueRequest.Builder requestBuilder = GetSecretValueRequest.builder().secretId(path);
     if (StringUtils.hasText(environmentProperties.getVersionStage())) {
	        requestBuilder.versionStage(environmentProperties.getVersionStage());
	}

I played with this since at Spring Cloud AWS secrets manager I am looking to add similar support for Spring.config.import we support.

[dantonyuk]

Hi @MatejNedic,

Thank you for sharing this!

As per my understanding, AWS Secrets Manager Secret versions and staging labels are not something that we can define. All version IDs are just UUIDs, and staging labels are just a set of predefined strings.

So I see these drawbacks:

1. There is no way to define custom labels or version IDs on our own. E.g. if we have two backends: git and awssecretsmanager, we have to tag commits with AWS staging labels' names. We lose the flexibility of naming the labels the way we want.
2. There is no way to fetch the secrets with the same version (except predefined staging labels). It limits us to have only three predefined spring cloud config labels. We lose the opportunity to have more than three labelled sets of configurations.
3. AWS cleans up old versions without asking. You can't have "dangling" versions, only the versions with staging labels. We lose old versions and can't have them back.
4. Staging labels are not the same as versions. If I want to compare the current release with the previous one I have to move the git tag along with the AWS Secrets Manager Secrets rotation. We lose the possibilities of having minor releases, or comparing the releases other than the current and the previous.

The former two do limit a lot the workflow we can have with the Spring Cloud Config. The latter two drastically affect version tracing and troubleshooting.

[MatejNedic]

Hi @dantonyuk ,

1. You can add any named staging label you want. Problem is when it is AWSCURRENT and AWS moves it to AWSPREVIOUS it will delete your label. One option to fight is that you don't label secrets that are AWSCURRENT and once you change it you Label it. One thing to note you can even set next versionID when you are doing the update of a Secret. I ran this against localstack works like a charm:

SecretsManagerClient secretsManagerClient = SecretsManagerClient.builder().region(Region.of(localstack.getRegion())).endpointOverride(localstack.getEndpointOverride(SECRETSMANAGER)).build();
GetSecretValueResponse getSecretValueResponse = secretsManagerClient.getSecretValue(GetSecretValueRequest.builder().secretId("/certs/prod/fn_certificate").build());
secretsManagerClient.updateSecret(UpdateSecretRequest.builder().secretId("/certs/prod/fn_certificate").clientRequestToken("6c155336-210f-43ee-9d2d-57a29a32e31c").secretString("my new string").build());
secretsManagerClient.updateSecretVersionStage(UpdateSecretVersionStageRequest.builder().secretId("/certs/prod/fn_certificate").versionStage("myrandomUUID").moveToVersionId(getSecretValueResponse.versionId()).build());
secretsManagerClient.updateSecret(UpdateSecretRequest.builder().secretId("/certs/prod/fn_certificate").clientRequestToken("6d155336-210f-43ee-9d2d-57a29a32e31c").secretString("my fake string").build());
getSecretValueResponse = secretsManagerClient.getSecretValue(GetSecretValueRequest.builder().secretId("/certs/prod/fn_certificate").versionStage("myrandomUUID").build());

clientRequestToken = new versionId.

2. Is not a problem anymore since you can have up to 20 defined different labels on a secret. Take a note what they say in Documentation:

You can also attach your own staging label to a version, for example to indicate development or production versions. You can attach up to 20 staging labels to a secret. Two versions of a secret can't have the same staging label.

Only problem now is that you are limited to 20 labels. This limits integration not gonna lie, but again when thinking about it do you really need a secret from a release that is no longer in production like 6 months old? I doubt that. If you do rollback you do it to previous version and delete a new version. Again don't know your use case and you will probably give me some better look on situation.

3. They start doing it if you pass 100 versions, I think 100 versions is a lot and you should not be using a secret from 90 versions ago anyway.

Secrets Manager removes outdated versions when there are more than 100, but it does not remove versions created less than 24 hours ago. If you update the secret value more than once every 10 minutes, you create more versions than Secrets Manager removes, and you will reach the quota for secret versions.

[dantonyuk]

Hmm... I did not know they allow defining custom labels. Thank you! Let me play with it.

[dantonyuk]

@MatejNedic, thank you!

I've implemented label support based on the staging labels.

[ryanjbaxter]

@dantonyuk sorry for not getting to this sooner.

One question I have is what happens if no label is provided and no default label is set. I tried to look through the tests to see if that was covered but I didn't see it, maybe I missed it.

[ryanjbaxter]

Polished this in https://github.com/spring-cloud/spring-cloud-config/pull/2233