Confusions about AKS secrets encryption at rest

[dhei]

Hi kubernetes kms team,

I have questions about AKS encryption at rest and hope to get some clarity on that.

We use Azure Key Vault Provider for Secrets Store CSI Driver on AKS to manage our secrets. Recently we noticed that the base64-encoded unencrypted kubernetes secrets can be accessed via kubectl commands or from azure portal. So I have done some reading on the topic and found confusing information.

• k8s official docs (link) says k8s secrets are unencrypted in etcd by default and recommends using EncryptionConfiguration with --encryption-provider-config flag.

• AKS docs (link) says Kubernetes secrets are stored in etcd, a distributed key-value store. Etcd store is fully managed by AKS and data is encrypted at rest within the Azure platform. This contradicts with the unencrypted secrets we saw from kubectl commands or from azure portal.

• This repo's README (link) mentioned AKS does encrypt secrets at rest, but keys are managed by the service and users cannot bring their own.

So my questions are:

1. What does the "encryption at rest" mean if we can see unencrypted secrets from kubectl and portal? Does it mean that secrets are encrypted at the disk level but authorized users to the AKS cluster would still be able to read the unencrypted secrets?
2. Does it mean that so we can’t bring our own encryption keys?

Thank in advance for any help or insights.

[nilekhc]

Hello @dhei,

What does the "encryption at rest" mean if we can see unencrypted secrets from kubectl and portal? Does it mean that secrets are encrypted at the disk level but authorized users to the AKS cluster would still be able to read the unencrypted secrets?

When you use kubectl to fetch secrets, they are decrypted and returned. The right way to check if secrets are encrypted is to query etcd. Try this.

Does it mean that so we can’t bring our own encryption keys?

Using Keyvault KMS plugin as encryption provider allows you to bring your own key and encrypt your secrets at rest in etcd.

[clauney]

@nilekhc We're being asked about encryption at rest of etcd for an internal security review of my team's applications hosted in AKS, and I ran across this thread in doing some research.

As far as I know, we cannot use the procedure you linked to query etcd, as we don't have access to the master nodes.

Am I mistaken? Or if not, is there some way to verify that the contents of etcd are encrypted?

[nilekhc]

@clauney You are right. etcd for AKS is managed component and will not be accessible to user.

@miwithro Could you confirm if the data in AKS etcd is encrypted at rest?

[miwithro]

@nilekhc with base64 yes. We are adding kms to AKS next month.

[nilekhc]

Thanks @miwithro.

@clauney Support for KMS as managed add-on on AKS is coming soon.

[clauney]

@nilekhc thank you for the into and info. @miwithro do you mean it's base64 encoded? My experience with base64 encoding is you can decode it with CLI / other tools...is there some other base64 scheme you are talking about that uses keys or something else? From my experience, security standards treat base64 encoded text as plaintext.

[miwithro]

@clauney etcd by default is encrypted with base64 encoding. So we are adding KMS to make it more secure, and until that is there we recommend to not put secrets in the cluster for the reason you eluded to.

[clauney]

@miwithro thanks for that additional info. We don't really need the ability to bring our own key; this is more about the need to just have the content of secrets be encrypted to where an attacker with filesystem access couldn't just harvest the etcd content from the filesystem and access the secrets. But if we have to use KMS integration to get that and then deal with the customer managed key thing, we could do that.

[clauney]

BTW - it would be better, IMO, to make this more clear in AKS documentation. Base64 encoding isn't encryption, in that it can be reversed by anyone without any other info like a key. It's also easily recognized - moreso if you find it in the context of an etcd db, since kube defaults to storing secrets that way. Anyway, if someone assumes their data is encrypted and relies on that as a control, makes such assertion to an auditor, etc. there could be ramifications for their organization.

I have seen source control for Microsoft documentation in the past, and if you think whoever reviews end user PRs would be amenable, I will look for where the AKS documentation is and try to come up with an update and submit a PR. If I'm remembering wrong or if that documentation isn't in open source scope, I can just drop it and focus on KMS for our own stuff.

[clauney]

@miwithro is there a plan to have AKS implement the aks-engine enableDataEncryptionAtRest feature described here? From a look at https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/, that does implement a process to encrypt secrets with a key, though it seems static; perhaps the Keyvault integration is needed for rotation, etc. Overall it's hard to map AKS features to aks-engine configs and capabilities, both current and roadmap state. But if KMS is the direction, we'll focus on that.

[clauney]

In any case, much thanks to @miwithro and @nilekhc for your engagement on this! I am glad to have a direction to explore soon.

[miwithro]

@clauney actually I recommend customers use CSI Secret Store https://docs.microsoft.com/en-us/azure/aks/csi-secrets-store-driver That externalizes the secrets, AKV can rotate the secrets, and it is an Azure supported solution. Once KMS comes out, I will update the AKS docs so that customers understand the differences between the two.

[clauney]

Thanks! We'll check that out too, especially the synced option. I was looking to limit latency / etc. so wanted to avoid direct hits to KV, and the sync described here still fetches on demand, but at least for secrets shared among pods / services within a namespace, it may give us what we want.

[nilekhc]

Hi @clauney couple more pointer..

• Secrets Store CSI will primarily mount secrets from AKV on tmpfs volume mount in pod. Sync is additional feature we provide, which syncs volume mounted content as k8s secrets. If you absolutely need to have your secrets as k8s secrets (for eg. NGINX ingress controller with TLS) using this along with KMS gives complete secure solution.

• wrt to AKS-Engine, we recommend using CAPZ instead. We are going to deprecate AKS-Engine soon.

[ahmad-hamade]

Is there any update on enabling etcd encryption along with key rotation support with KMS? Other providers like AWS EKS have implemented the same almost 2 years ago

[miwithro]

@ahmad-hamade We will have a Public Preview within the next few weeks of KMS. This will not include key rotation support, as that is targeted for GA.

[LiorAlafiArmo]

@miwithro base64 is not encrypting anyone can do https://www.base64decode.org/ or w/e to decrypt your secret this is far from being encrypted that's like having it in plaintext...

[miwithro]

@LiorAlafiArmo we are releasing KMS etcd encrytion for AKS in Public Preview in the next few weeks as I eluded too above. I will refactor the document at that time.

[miwithro]

https://docs.microsoft.com/en-us/azure/aks/use-kms-etcd-encryption

[BojanOro]

@miwithro Is there a planned timeline or roadmap for etcd encryption GA release?

[miwithro]

Tentatively Aug 2022.

[lireanne]

Hi @miwithro, thanks for your help in the thread so far. I'm trying out the new preview KMS encryption feature by following the steps in the documentation you provided above, however at the last step "Update an exiting AKS cluster to enable KMS etcd encryption" I encountered the following error:

Your cluster is already using systemassigned managed identity, and you are going to update the cluster to use userassigned managed identity.
Are you sure you want to perform this operation? (y/N): y

(BadRequest) KeyVault network access is unspecified for Azure Key Vault KMS.
Code: BadRequest
Message: KeyVault network access is unspecified for Azure Key Vault KMS.

The specific command I used was az aks update --name <cluster-name> --resource-group <resource-group-name> --enable-managed-identity --assign-identity $IDENTITY_RESOURCE_ID --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID .

Hoping to get some further info on this error message as I was unable to find any documentation/discussion on it. Thanks so much in advance.

[miwithro]

@lireanne I will add a note to the document about this.

Please run az aks update --name <> --resource-group <> --enable-managed-identity --assign-identity $IDENTITY_RESOURCE_ID

Then run az aks update --name --resource-group --enable-managed-identity --assign-identity $IDENTITY_RESOURCE_ID --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID

[lireanne]

Thanks @miwithro , I tried that but unfortunately still it returned (BadRequest) KeyVault network access is unspecified for Azure Key Vault KMS.

[miwithro]

@lireanne Are you using Keyvault with Private Link? If so, we just checked that code in and it will be available in a few weeks.

[lireanne]

@miwithro This is an initial POC so I actually did not configure private link access to the KV. But thank you for the info, will try again later with a private link when possible.

[RichardChen820]

@miwithro I encounter same issue with this, and I didn't configure Private link to my keyvault. Do you have any idea on this?

Hi @miwithro, thanks for your help in the thread so far. I'm trying out the new preview KMS encryption feature by following the steps in the documentation you provided above, however at the last step "Update an exiting AKS cluster to enable KMS etcd encryption" I encountered the following error:

Your cluster is already using systemassigned managed identity, and you are going to update the cluster to use userassigned managed identity.
Are you sure you want to perform this operation? (y/N): y

(BadRequest) KeyVault network access is unspecified for Azure Key Vault KMS.
Code: BadRequest
Message: KeyVault network access is unspecified for Azure Key Vault KMS.

The specific command I used was az aks update --name <cluster-name> --resource-group <resource-group-name> --enable-managed-identity --assign-identity $IDENTITY_RESOURCE_ID --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID .

Hoping to get some further info on this error message as I was unable to find any documentation/discussion on it. Thanks so much in advance.

[lireanne]

Hi @miwithro and @nilekhc , revisiting this thread with some additional questions. Please provide guidance as I'm sure we're all concerned about the security of our secrets hosted on Azure. Thanks in advance.

I followed the documentation here - Set up Secrets Store CSI Driver to enable NGINX Ingress Controller with TLS - to sync an AKV certificate to a cluster. As expected, this created a secret, however when accessed with kubectl it's still returning as unencrypted data (b64-encoded only).

According to Azure documentations and earlier replies, this is because "the secret is decrypted when fetched by kubectl". But this takes me back to square one in this thread, where it appears no one can confirm whether the secrets are encrypted or not at rest. Ie. even though the certificate is encrypted in the AKV, its security may still be potentially exposed once synced into the k8s cluster?

Could you clarify:

1. The encryption provided for external CSI Secret Store/AKV secrets, is separate from the encryption for etcd/kubernetes secrets.
2. Is etcd data already encrypted at rest by Azure? Not talking about base64-encoding but actual encryption.
3. If no to above, should we all be implementing the KMS etcd encryption feature that's currently in preview? Would that return decrypted or encrypted data with kubectl?

[nilekhc]

1. The encryption provided for external CSI Secret Store/AKV secrets, is separate from the encryption for etcd/kubernetes secrets.

Secrets Store CSI driver today does not have built in encryption.

2. Is etcd data already encrypted at rest by Azure? Not talking about base64-encoding but actual encryption.

@miwithro could you confirm this?

3. If no to above, should we all be implementing the KMS etcd encryption feature that's currently in preview? Would that return decrypted or encrypted data with kubectl?

Yes. We recommend using KMS etcd encryption to protect Synced K8s Secrets. It would still return decrypted data with kubectl get, but data in etcd will be encrypted with KEK in Key Vault.

[ritazh]

Just FYI, the https://docs.microsoft.com/en-us/azure/aks/use-kms-etcd-encryption feature on AKS is now general available.