Custom resources don't get encrypted in etcd

[fritzduchardt]

What happened:

Trying to encrypt a custom resource in etcd with the following EncryptionConfiguration in kube-apiserver does not lead to encrypted state in etcd.

In this example the custom resource is named "tokens" and belongs to the group "management.cattle.io":

`apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources:

• providers:
  • aescbc: keys:
    • name: some-key secret: some-secret
  • identity: {} resources:
  • tokens.management.cattle.io
• providers:
  • aescbc: keys:
    • name: some-key secret: some-secret
  • identity: {} resources:
  • secrets`

What you expected to happen:

Expecting custom resources to be encrypted in etcd if queries e.g. like this: etcdctl get /registry/management.cattle.io/tokens/some-token

How to reproduce it (as minimally and precisely as possible):

Activate encryption on the kube-apiserver with a EncryptionConfiguration definition on any custom resource and query state in etcd.

Anything else we need to know?:

Tested this both with namespaced and non-namespaced custom resources. In both cases encryption does not work.

Environment:

• Kubernetes version: v1.20.6
• Cloud provider or hardware configuration: Openstack
• OS: flatcar 2765.2.3
• Kernel: Linux host-10-13-39-90 5.10.32-flatcar #1 SMP Tue Apr 27 22:38:30 -00 2021 x86_64 Intel Core Processor (Haswell, no TSX, IBRS) GenuineIntel GNU/Linux
• Install tools: rke version v1.2.9

[neolit123]

/sig api-machinery

[neolit123]

/sig auth

[neolit123]

In this example the custom resource is named "tokens" and belongs to the group "management.cattle.io":

can you please format the example properly with ```

are you seeing any complains from the kube-apiserver about the given configuration?

[caesarxuchao]

/remove-sig api-machinery please re-tag us if sig auth needs support from sig apimachinery on this issue.

[enj]

Adding back API machinery since this is a known issue with the wiring of the createAPIExtensionsConfig. It drops the RESTOptionsGetter from KAS:

https://github.com/kubernetes/kubernetes/blob/6a043332bed986be691f9446c6b97ce3e7217c64/cmd/kube-apiserver/app/apiextensions.go#L53

But then fails to account for EncryptionProviderConfigFilepath which would set TransformerOverrides (other aspects of the storage config may also be accidentally ignored). Perhaps these code paths should be refactored to better re-use the same logic in all places.

/sig api-machinery /triage accepted

[enj]

Note that there has been discussion in the past on how best to handle encryption at rest for resources that are dynamically created via CRDs. i.e. having to provide static config to KAS for some unknown future resource does not really fit the model. IIRC there was some desire to be able to mark a CRD as "sensitive" to indicate that it needs encryption at REST. This would then be dynamically honored by the underlying storage layer and possibly exposed in discovery metadata. Such a change would likely require a KEP.

[enj]

Based on discussion during the 2021-08-18 SIG Auth meeting, the current proposal is to fix the immediate issue in the code outlined in https://github.com/kubernetes/kubernetes/issues/104165#issuecomment-895273751 - i.e. support encryption of resources dynamically created via CRDs statically via EncryptionConfiguration.

If someone wants to write a KEP to handle this dynamically, I think there is interest in that functionality, but no such work is currently planned.

Andrew has volunteered to create the bug fix along with supporting integration tests.

/assign @ankeesler

[ankeesler]

Reproducer script: https://gist.github.com/ankeesler/d1369b0d96c2a071432e617d5eb5f56b WIP PR (not currently working, just posting status): https://github.com/kubernetes/kubernetes/pull/104662

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[enj]

/remove-lifecycle rotten /ifecycle frozen

[enj]

/lifecycle frozen

[enj]

/unassign @ankeesler /assign @ritazh cc @aramase

Rita wanted something difficult to work on 😄

See #104662 and the other comments I have made in this issue.

[enj]

/milestone v1.26

[HeshamAboElMagd]

Hello @ritazh , just wanted to touch base for any updates regarding this issue. Thank you!

[ritazh]

Working on it! :)