CoreDNS PreemptionPolicy conflicts with host cluster global default Priority Class

[ruifung]

What happened?

When trying to deploy a vcluster to my homelab cluster, I found out that the coredns pod fails to sync because it has a preemptionPolicy set that is different from the default PriorityClass on my cluster.

The syncer fails to sync the coredns pod with the following error

syncer 2024-04-25 04:11:12    ERROR    controller/controller.go:329    controller pod: namespace kube-system name coredns-74987d4979-k272p: reconcileID "a0a6be23-8e5e-45bb-886c-414074d250d9": Reconciler error pods "coredns-74987d4979-k272p-x-kube-system-x-vcluster-rf-ebceb13e69" is forbidden: the string value of PreemptionPolicy (PreemptLowerPriority) must not be provided in pod spec; priority admission controller computed Never from the given PriorityClass name    {"component": "vcluster-pro"}

What did you expect to happen?

For it to successfully sync the pod. Ideally by omitting the preemptionPolicy here. Or mapping it properly to the host's system-cluster-critical PriorityClass.

How can we reproduce it (as minimally and precisely as possible)?

1. Have a PriorityClass with globalDefault set to true with a preemption policy set to Never
2. Deploy vcluster with default settings
3. Observe syncer failing to sync coredns pod

Anything else we need to know?

After temporarily removing the globalDefault PriorityClass on the host cluster, the pod successfully syncs, and I've observed that the resulting pod has an empty priorityClassName, which causes the host cluster to apply the default priorityClass which conflicts with the pod that has the values of system-cluster-critical inside the vcluster, but not actually having that priorityClassName.

Host cluster Kubernetes version

$ kubectl version
Client Version: v1.29.2                                                                                        
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3                                                        
Server Version: v1.29.3    

Host cluster Kubernetes distribution

k8s

vlcuster version

$ vcluster --version
vcluster version 0.20.0-beta.1

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

k8s

OS and Arch

OS: Talos Linux 1.7.0
Arch: x64

[narcis96]

Hello

Thank you for the information regarding the issue you are facing. From the error message provided it seems that there is a conflict, between the preemptionPolicy of the coredns pod and the default PriorityClass set in your host cluster.

This conflict arises because the default PriorityClass of your host cluster has a preemption policy of 'Never' which does not align with the 'PreemptLowerPriority' policy specified in the coredns pod.

At my investigation it may be either a miss-configuration or a real problem close to https://github.com/loft-sh/vcluster/blob/v0.20.0-beta.1/pkg/controllers/resources/priorityclasses/translate.go#L26.

I would recommend to provide any additional logs or configurations that might help our team replicate and address this issue.

[ruifung]

What kind of additional logs and/or configurations would be helpful here?

The default preemption policy on my host cluster is indeed intended to never preempt other pods.

As previously stated, I'm using a default configuration of vcluster as deployed via CLI, so I'm not entirely sure how this can be a misconfiguration if the default configuration is producing pod specs that is conflicting with a host cluster default priorityClass preemption policy.