vcluster

vcluster copied to clipboard

What happened?

When using the --cluster-domain command line parameter to set a custom clusterDomain, this is passed down to the kubelet and certificates but the coredns configuration does not change. This leads to failing lookups inside the cluster using that customDomain.

What did you expect to happen?

The coredns configMap should be updated accordingly. Currently its hardcoded, but it should be dynamic like in the k3s coredns configmap

How can we reproduce it (as minimally and precisely as possible)?

start a vcluster with --custom-domain=my.domain.local and try a ping kube-dns.kube-system.svc.my.domain.local in any pod

Anything else we need to know?

There's https://github.com/loft-sh/vcluster/pull/287 allowing to add custom imports to the coredns file but this doesn't help as you can't override the existing block.

Host cluster Kubernetes version

does not apply

Host cluster Kubernetes distribution

does not apply

vlcuster version

0.6.0

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

all, as the coredns configmap is defined by vcluster

OS and Arch

any

Hello @an-tex :wave: Thank you for reporting this problem. I don't think this is a bug, I'll treat this as a feature request. For now, we will hold off on implementing the required code changes (new CLI flag for syncer, etc.), but we will reconsider if more people are requesting this.

But I can offer you a workaround that should resolve your problem. The coredns.yaml manifest used by vcluster can be customized by mounting your own file its path in the syncer container - /manifests/coredns/coredns.yaml. In the host namespace where vcluster is installed you need to create a ConfigMap that will contain your modified coredns.yaml manifest. Next, add the necessary volumeMount entries into the .syncer.volumeMounts + volume entries into the .volumes (or .syncer.volumes for k8s distro) in the helm values for your vcluster. Some k8s docs about this - https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#add-configmap-data-to-a-specific-path-in-the-volume I hope this workaround will work for you. I am happy to help if any issue arises.

Hi @matskiv , thanks for the quick reply. Your workaround works like a charm! :)

as a note: in addition to k3s, k0s has also just added a configuration parameter for that in https://github.com/k0sproject/k0s/pull/1569 .

the workaround got actually easier with vcluster 0.7 as the coredns file can be directly specified in the helm values

@FabianKramm tbh i'm not sure this can be closed. the #867 PR only addressed the part where it's possible to pass the clusterDomain to k0s. the issue with the hardcoded cluster.local in the coredns still remains (even though there's the mentioned workaround). i guess ideally this would be fixed by applying the same cluster domain value in the coredns template. i could looking into creating a follow PR handling that (at least for k0s)

@an-tex I agree, so I'll reopen this. Thank you for the PR, we will try to get to it as soon as we can.