[backend] kubernetes: fix secret size limitation

[k0da]

By now kubernetes backend could hold up to defaultETCDSize gzipped data (which is 1-1.5Mb). This doesn't scale for larger states.

This commit implements spliting data across multiple secrets bound by the same Secret labels. This practically removes etcd value size limitation and allows backend to scale across multiple secrets.

This also takes care of cases when state needs to be shrinked. In such case code will cleanup unneeded secrets.

Signed-off-by: Dinar Valeev [email protected]

[Skaronator]

This is basically a similar fix as https://github.com/hashicorp/terraform/pull/28838, but for the kubernetes state file.

[hashicorp-cla]

All committers have signed the CLA.

[crw]

Thanks for this submission! I will raise this with the internal maintainers of the kubernetes backend. Although I cannot commit to having this PR reviewed at this time, we acknowledge your contribution and appreciate it! Thanks again for the submission!

[Bernix01]

Just making noise to this PR, is this a ready to go?

[jrhouston]

Also, a more general question: is there potential for conflict if more than one terraform workspace is stored in the same K8s namespace?

The purpose of the secret_suffix attribute here is so you can store multiple terraform states in the same namespace and the secrets will get a different name. You would cause problems if you deliberately used the same prefix across more than one terraform config.

[Bobonium]

Was there a specific reason why this was not merged and released after @alexsomesan approved it in february? This feature would still be very much appreciated.

[crw]

Looks like it got missed. I will add it to the triage queue. Thanks!

[kmoe]

Thanks for your patience, everyone!

[github-actions[bot]]

Reminder for the merging maintainer: if this is a user-visible change, please update the changelog on the appropriate release branch.

[sftim]

When the data are gzip encoded, what I'd expect to find is that each Secret holds a valid gzip stream. Concatenating these also produces a gzip stream that you can uncompress and deserialize.

Is that how it works?

[sftim]

:thought_balloon: I wonder whether and how we provide atomic updates when there are multiple Secrets

[github-actions[bot]]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.