aws-quickstart
Secret Store Add-on: Problem reading key/value kind of secret from AWS Secret Manager
Describe the bug
We have a helm chart configuration that reads the secrets mounted on to K8s cluster. Using EKS blueprints we created the nodes/pods from those helm charts on an EKS cluster. The helm charts expect the secret name to be my-secret-name and look for different keys in different chart definitions across the project like my-secret-key-1, my-secret-key-2, etc.,
All these keys and values (secrets) are saved in AWS Secret Manager like
{
"my-secret-key-1": "key-1-value",
"my-secret-key-2": "key-2-value",
...
}
Note: The code snippet of EKS Blueprints and helm charts is in the Reproduction Steps section. Please check
The secret team, secret provider class, cluster, etc., gets created and the secret also gets mounted with this setup. The problem exists in the way/format of the secret that got mounted.
Expected Behavior
When the kube config is updated and kubectl get secret -n my-namespace command is run, the output is expected to have the
NAME TYPE DATA AGE
default-token-rx4zp kubernetes.io/service-account-token 3 5d22h
my-secret-name Opaque 37 44h
37 is the number of keys we have in secret
And when you describe the secret all the keys should be listed like
Name: my-secret-name
Namespace: my-namespace
Labels: cattle.io/creator=norman
Annotations: field.cattle.io/creatorId: u-kqxyaw6tcm
...
Type: Opaque
Data
====
my-secret-key-1: 26 bytes
my-secret-key-2: 12 bytes
...
...
And when the secret is read in helm charts, it should just return the value of that secret key.
Current Behavior
kubectl get secret -n my-namespace is returning
NAME TYPE DATA AGE
default-token-rx4zp kubernetes.io/service-account-token 3 5d22h
my-secret-name Opaque 1 44h
The issue here is that all the key/value items are coming as one object (stored as JSON) as you see in the DATA column of first out above and the kuebctl describe secret/my-secret-name -n my-namespace is returning only 1 item.
Name: my-secret-name
Namespace: my-namespace
Labels: cattle.io/creator=norman
Annotations: field.cattle.io/creatorId: u-kqxyaw6tcm
...
Type: Opaque
Data
====
my-secret-key-1: 2677 bytes
If we have multiple items in the data section of the CDK code (TeamApp class above), the describe secret command, the result will repeat those many items with the same size (2677 bytes). See below
Name: my-secret-name
Namespace: my-namespace
Labels: cattle.io/creator=norman
Annotations: field.cattle.io/creatorId: u-kqxyaw6tcm
...
Type: Opaque
Data
====
my-secret-key-1: 2677 bytes
my-secret-key-2: 2677 bytes
Reproduction Steps
export class BluePrintsCluster extends Stack {
constructor(scope: Construct, id: string, props: DeployStackProps) {
super(scope, id, props);
const clusterName = `${props.stackName}-learning`;
const repoUrl = 'https://github.com/Organization/helmcharts-repo.git';
const addOns: Array<blueprints.ClusterAddOn> = [
new blueprints.addons.VpcCniAddOn(),
new blueprints.addons.SecretsStoreAddOn(),
new ArgoCDAddOn({
bootstrapRepo: {
repoUrl,
credentialsSecretName: "github-token",
credentialsType: "TOKEN",
targetRevision: 'branch-name',
path: "apps",
},
adminPasswordSecretName: "adminPasswordSecretName",
}),
new blueprints.addons.AwsLoadBalancerControllerAddOn(),
new blueprints.addons.EfsCsiDriverAddOn(),
];
blueprints.EksBlueprint.builder()
.addOns(...addOns)
.account(props.env?.account)
.region(props.env?.region)
.teams(new TeamApp(this))
.build(this, clusterName);
}
}
export class TeamApp extends ApplicationTeam {
constructor(scope: Construct) {
super({
name: 'app',
namespace: 'my-namespace',
teamSecrets: [
{
secretProvider: new blueprints.LookupSecretsManagerSecretByName(
'my-secret-name',
),
kubernetesSecret: {
secretName: 'my-secret-name',
type: blueprints.KubernetesSecretType.OPAQUE,
data: [
{
key: 'my-secret-key-1'
}
]
},
},
],
});
}
}
Part of the helm chart code
# Secrets spec definition
serviceAccountName: app-sa
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: app-aws-secrets
# mounting secrets-store as a volume
volumeMounts:
- name: secrets-store-inline
mountPath: /mnt/secrets-store
readOnly: true
// secret is read like
customSecret:
SECRET_1:
my-secret-name: my-secret-key-1
Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.37.1
EKS Blueprints Version
1.2.0
Node.js Version
16.6.0
Environment details (OS name and version, etc.)
Linux
Other information
No response
Hello @jagadeeshmaneri , my apologies for not replying sooner.
The reason why this was missed is because it does not appear to be a blueprints issue. Rather it is a feature of CSI Secret Store driver.
My understanding is that you have a multi-key secret in AWS and you are trying to produce a multi-key secret in EKS. The CSI Secret Store driver is very flexible, to the point when it is not quite intuitive. It was designed to be able to pull a single secret from AWS and map it to several secrets in Kubernetes or a single secret in different configurations.
Here is the shape of a SecretStoreProviderClass that creates a multi-key secret in EKS:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aws-secrets
spec:
provider: aws
parameters:
objects: |
- objectName: "arn:aws:secretsmanager:us-east-2:111122223333:secret:MySecret-a1b2c3"
jmesPath:
- path: username
objectAlias: dbusername
- path: password
objectAlias: dbpassword
Please refer to the official documentation for more info.
When defining team secrets with EKS Blueprints you can use the jmesPath array to provide mapping between keys in the AWS Secret and Kubernetes secret. Example below takes a secret with three keys (username, password, url) and maps them to a secret in k8s
return {
secretProvider: new LookupSecretsManagerSecretByName(secretName),
jmesPath: [{ path: "url", objectAlias: "url" }, { path: "username", objectAlias: "username" }, { path: "password", objectAlias: "password" }],
kubernetesSecret: {
secretName: secretName,
labels: {"argocd.argoproj.io/secret-type": "repo-creds"},
data: [
{ key: "url", objectName: "url" },
{ key: "username", objectName: "username" },
{ key: "password", objectName: "password" }
]
}
};
Note that the data attribute is also very important. The objectName in the data mapping corresponds to the objectAlias in jmesPath. Hope it helps, and I agree that this particular behavior may not be the most evident to the customers.
