Encrypt with multiple public keys (disaster recovery, ...)

[mkmik]

Proposal

Teach kubeseal to emit multiple versions of the same encrypted item, each one encrypted with a different cert.

Rationale

There are two major use cases:

1. disaster recovery
2. multi-cluster / multi-env

Disaster recovery

Since the introduction of automatic key renewal in #137, the process around disaster recovery become more complex since now users have to back up the sealing keys matching the label sealedsecrets.bitnami.com/sealed-secrets-key=active relatively frequently. If the cluster blows up after a new key has been created and before a backup has happened, all recently sealed secrets will be effectively lost.

If kubeseal can seal with multiple certificates, users could have a "backup key pair", whose private key is securely stored in a (physical?) vault and used only during

Decoupling the disaster recovery procedure from the creation of new sealing keys will allow us to explore more interesting patterns. For example, we could create a new private key per namespace, or even a new private key for each sealed secret! The trade-off between online/offline sealing still remains, but removing the disaster recovery process makes it more palatable to explore operation modes that involve multiple secrets created at arbitrary cadences.

Multi-cluster / multi-env

Ideally secret should be as fine grained as possible in order to reduce the impact of a leak and subsequent secret revocation. E.g. if you need a slack API key in a bunch of applications, possibly running in different clusters, it's better to create a token for each application and seal it independently.

That said, in some cases that's not practical or perhaps not even possible and you end up sealing the same underlying secret into multiple sealed secrets (or using cluster-wide scope if applicable)

Instead of scattering around each of those sealed secrets in independent files, we could use the "multi-encryption" feature to have a single sealed secret that seals the same items for N target environments (either different namespaces of the same cluster or even different clusters altogether)

References

• #120
• #226

[tshak]

I would focus instead on integrating with 3rd party key management solutions such as:

• AWS KMS
• Hashicorp Vault
• Google KMS

This will solve the disaster recovery scenario while also being more secure by default as well as easier to audit. Since this could be a lot of effort I'd just pick one (maybe AWS KMS since it's probably the most popular), and make it easy for the community to contribute other integrations.

See also: https://github.com/bitnami-labs/sealed-secrets/issues/64

[mkmik]

yeah, I also think integrating with "KMS" services is a low hanging fruit going to help out many people. We hear you!

This proposal is more for those people who don't have access to that infrastructure. (yes I put Hashcorp Vault in the infrastructure bin, because if it's just about running hashcorp vault in the very cluster you're running your sealed secrets, we're back to square one)

[tshak]

The challenge here is that people who don't have access to any form of secure credential storage outside of k8s are not likely to be equipped to securely store a backup key. In my opinion the only legitimate places to store the backup key would be via a KMS style service or with software like Vault.

As an aside, I wouldn't put Hashicorp Vault in the list because the assumption would be that it would be setup with disaster recovery. It's designed exactly for this purpose: the long term storage of secrets.

P.S. Good discussion and thanks for the awesome project! I haven't used it in production yet but I find this type of solution for secrets management much easier than many other approaches.

[Datamance]

I would focus instead on integrating with 3rd party key management solutions such as:

• AWS KMS
• Hashicorp Vault
• Google KMS

This will solve the disaster recovery scenario while also being more secure by default as well as easier to audit. Since this could be a lot of effort I'd just pick one (maybe AWS KMS since it's probably the most popular), and make it easy for the community to contribute other integrations.

See also: #64

• 10000 to this! Are there any current workarounds for using something like GKMS with Sealed Secrets? That would be ideal - instead of using something like cat mysecret.json | kubeseal | jq -r .spec.data > mysec-ssdata.txt to get/write the encrypted data, I could use the google cloud client libraries that are already a part of my toolchain.

[mkmik]

@Datamance sorry I don't understand; what problem are you trying to solve?

[Datamance]

@mkmik if SealedSecrets were able to integrate with different backends, then (ostensibly) you could decouple the cluster-side components (Controller and CRDs from what I understand) from the client-side component (the kubeseal tool).

Right now I’m trying to puppet everything from Python, using the jsonnet bindings and google cloud client libraries (notably, gKMS for encryption/decryption). The kube.libsonnet library has a nice helper for creating SealedSecret objects. Rather than having my tooling shell out to kubeseal just to author a SealedSecret, if I could set up the SealedSecret Controller to use my gKMS crypto keys in the first place, then in my tooling I could just rely on the python gKMS client to pass encrypted keys to the jsonnet templating machinery (which uses that SealedSecret jsonnet helper I mentioned earlier).

Hope that helps - let me know if anything needs clarification.

[Datamance]

@mkmik friendly ping - does what I'm trying to do make sense?

[ryh-cc]

any chance this could be bumped for AWS KMS support? :)

[mkmik]

What do you mean?

[Datamance]

@mkmik who is that question for?

[mkmik]

@Datamance sorry I missed your thread; will answer ASAP

@ryh-cc what do you mean?

[mkmik]

Delegating crypto to an external KMS (e.g. aws or gcloud) is certainly useful and I'd love to support that. For one, it does move the problem of disaster recovery to some place else.

However, this proposal is still useful since it would allow to use the same sealed secret resources with multiple clusters that don't share the same cloud provider.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[hazcod]

Unstale.

[crstian19]

Some news about KMS support? I think it would be a change that would be good for a lot of people

[reegnz]

I've opened an issue about offloading crypto a while ago, KMS is definitely part of that: https://github.com/bitnami-labs/sealed-secrets/issues/779

[crstian19]

Some news about KMS support? I think it would be a change that would be good for a lot of people

I found this: https://github.com/EladLeev/kubeseal-convert#aws-secrets-manager

[reegnz]

@crstian19 I think the goal of this ticket is not to migrate to go-crypto, but to actually allow using KMS, Vault, or any other managed crypto service. Some of us have security requirements that don't allow us to use crypto-tools that are not FIPS-compliant (eg. go-crypto).

So the requirement is almost the reverse of what you've linked. :) (not exactly, I'd still want encrypted secrets to be in git instead of SecretManager, but encrypted with KMS instead of go-crypto).

[frmrm]

However, this proposal is still useful since it would allow to use the same sealed secret resources with multiple clusters that don't share the same cloud provider.

I just want to +1 this statement. Using Sealed Secrets in a multi-region active/active or active/passive environment is a bit tedious right now. If we could encrypt against multiple public keys and produce a single ciphertext that would simplify some operational burden for things I need to do.