sealed-secrets
sealed-secrets copied to clipboard
bitnami-labs
Decrypt performance: add workload sharding for replicas > 1
Summary
Add decrypt workload sharding split by replica.
Motivation
Currently there is no explicit support for multiple replicas (discussions in https://github.com/bitnami-labs/sealed-secrets/issues/62 but the issue is dead). This means that there are limitations in scalability when there are hundreds to thousands of secrets that need regular decryption.
Example: we have thousands of sealed secrets, when we add a new sealed secret it takes an unknown time to decrypt (if at all in some cases). We want to scale to meet the processing needs of these cryptographic functions.
Proposal
Kubeseal instance should calculate workload based on total targeted sealed-secrets and it's replica count.
Context / Environment
kubernetes v. 1.18.15 kubeseal v. 0.12.16 Sealed secrets / instance: 500-4000
This would be nice to have indeed. How would each replica know who it is?
(I wanted to avoid StatefulSets due to bugs like https://github.com/kubernetes/kubernetes/issues/67250 still being unsolved)
It may be a coincidence, but such vocal response to a feature request makes me wonder if we didn't hear a major source of pain the community was having.
Is everybody here actually having problems caused by sealed-secrets controller not being able to decrypt stuff fast enough?
In my case we are having a gitops workflow where we treat applications configuration as secrets (contains password etc...). It used to be fine but now we reached a point were we have over a 2000 encrypted secrets and some times decrypting the secret even fails. The worst part is when we do maintenance on the cluster / rotate the key, the time needed to re-decrypt everything is close to an hour and often fails for some of the secrets that we then need to chase and delete / recreate to trigger decryption.
My impression is that kubeseal does things one-by-one, which is fine as long as the balance of updates vs number of secrets stays ok. But even in the 500-1000 sealed-secrets zone it takes long enough that some secrets are sometimes never processed for us.
This would be nice to have indeed. How would each replica know who it is?
(I wanted to avoid StatefulSets due to bugs like kubernetes/kubernetes#67250 still being unsolved)
Interestingly the guys at argoproj have recently added horizontal scaling based on replica counts, perhaps their code code serve as inspiration? They also need to create execution queues based on K8S objects:
- https://github.com/argoproj/argo-cd/blob/release-1.8/controller/sharding/sharding.go
- https://github.com/argoproj/argo-cd/pull/4285/files
- https://github.com/argoproj/argo-cd/blob/release-1.8/controller/sharding/sharding.go
it's not the greatest thing, but it's at least functional, you set the replicas, then an environment variable that's passed to the program so it can calculate it independently.
We also reach a breaking point on our dev where the number of secrets is pretty large and when we roll out a new dev cluster it takes more than 30 minutes to decrypt everything
