kubermatic
API fails to fetch `EtcdRestores` (and other resources?) in dedicated master/seed architecture
What happened
When setting up a dedicated master/seed architecture with KKP 2.22.2 (or the latest release branch commits for release/v.22), the UI returns errors on the project overview page. The kubermatic-api pods involved log the following error:
{"level":"debug","time":"2023-05-12T12:21:42.319Z","caller":"kubermatic-api/main.go:599","msg":"response","body":"{\"error\":{\"code\":500,\"message\":\"no matches for kind \\\"EtcdRestore\\\" in version \\\"kubermatic.k8c.io/v1\\\"\"}}\n","status":500,"uri":"/api/v2/projects/f465kk5nv6/etcdrestores"}
@pkprzekwas and I have been debugging this, and we believe this is happening due to the shared REST mapper injected into all providers as the error is thrown by discovery involving the REST mapper in question:
https://github.com/kubermatic/dashboard/blob/c0c82888a964be7248148c37670f83db132f604a/modules/api/cmd/kubermatic-api/main.go#L349
This shared REST mapper utilises the master cluster for information about resources, even when the clients generated with a reference to the REST mapper are for seeds. That decision made sense when it was made - all CRDs lived on both master and seed. Unfortunately, this assumption was invalidated by https://github.com/kubermatic/kubermatic/pull/10903, which was shipped starting with KKP 2.22.0.
As a note: Usage of the master REST mapper was also a problem in kubermatic/kubermatic, e.g. as described in https://github.com/kubermatic/kubermatic/pull/12188.
This is kind of part of https://github.com/kubermatic/kubermatic/issues/12186.
Expected behavior
The API is capable of listing EtcdRestores and all other resources that only live on seed clusters.
How to reproduce
- Set up KKP 2.22 with separate/dedicated master and seed clusters
- Create a Project from the dashboard
- View that project; the error message should happen
Environment
- UI Version: v2.22.2
- API Version: v2.22.2
- Domain: n/a
- Others:
Current workaround
Apply all CRDs to master cluster.
Affected user persona
Users with large scale KKP setups as described in our documentation.
Business goal to be improved
Metric to be improved
Note: I'm not sure we can fix this on short notice, I've started a thread to discuss if we want to fix this here on the dashboard side or on the kubermatic/kubermatic side.
While we fixed this by bringing back all CRDs, I believe it still makes sense to progress on decoupling. As it stands, this issue is a prerequisite for bringing back the CRD split.
Issues go stale after 90d of inactivity.
After a furter 30 days, they will turn rotten.
Mark the issue as fresh with /remove-lifecycle stale.
If this issue is safe to close now please do so with /close.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
/lifecycle rotten
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
/close
@kubermatic-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity. Reopen the issue with
/reopen. Mark the issue as fresh with/remove-lifecycle rotten./close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.