out-kube-config-secret-namespace feature produces unknown namespace for the cache error

[guyguy333]

What happened?

I try to use out-kube-config-secret-namespace syncer feature but I'm getting this error:

Error writing kube config to secret: creating xxxxx-kubeconfig secret in the vclusters ns failed: apply generated kube config secret: unable to get: vclusters/xxxxx-kubeconfig because of unknown namespace for the cache

vCluster syncer is running in vcluster-xxxx namespace and I would like my kubeconfig secret to be available in vclusters namespace. I added this RBAC so syncer pod can read/write secrets in vclusters namespace without luck. Any idea what's wrong ?

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: vclusters-kubeconfig-rw
  namespace: vclusters
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: vclusters-kubeconfig-rw-xxxxx
  namespace: vclusters
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: vclusters-kubeconfig-rw
subjects:
  - kind: ServiceAccount
    name: vc-vcluster
    namespace: vcluster-xxxx

What did you expect to happen?

• No longer syncer error and a secret write in the right namespace
• Minimal RBAC documented for this feature

How can we reproduce it (as minimally and precisely as possible)?

Just use out-kube-config-secret-namespace feature with a namespace different than vcluster namespace

Anything else we need to know?

No response

Host cluster Kubernetes version

$ kubectl version
1.28

Host cluster Kubernetes distribution

EKS

vlcuster version

$ vcluster --version
v0.16.2

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

# Write here

OS and Arch

OS: 
Arch:

[rikycaldeira]

I've hit this problem as well, seems that in the current version of controller-runtime it is entering multi-namespace mode, and when creating the blocking cache client if the separate kubeconfig secret namespace is not in the list of default namespaces no namespace cache mapping is initialized

I'd suggest creating a new client to interact with the separate namespace to manage the kubeconfig secret, unless there's a way to do it via the existing blocking cache client? If the separate namespace is added to the default namespaces of the existing client it probably will initialize informers for a lot more than the kubeconfig secret

[matskiv]

@rikycaldeira sounds like you understand the problem quite well, would you be interested in contributing a PR? And since it is Hacktoberfest, Loft is also giving away some swag for the PR contributors. :)

[rikycaldeira]

I can give it a try, although I'm not super familiar with the codebase, hope I don't ruin something :D I'll follow my own suggestion of instantiating a dedicated client and put it up for review

[matskiv]

Awesome! Feel free to reach out to the team in Slack if you need any help. Don't forget to check out contributing.md, we are using DevSpace for dev, which should make iterating the implementation easier :)

[guyguy333]

@rikycaldeira @matskiv Did you work on a fix ? Thanks :)