Make default `pods.spec.tolerations` customizable

[ababila]

Is your feature request related to a problem?

Yes. When pods, for example for debugging or as part of a DaemonSet, are created within a vCluster, tolerations such as those in the list below are systematically added to the pod's specification. These automatically-added tolerations may be blocked by policies enforced by the cluster administrator, causing the pods to be not synchronizable on host cluster.

...
Controlled By: DaemonSet/my-daemonset 
...
Tolerations:    node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                node.kubernetes.io/not-ready:NoExecute op=Exists
                node.kubernetes.io/pid-pressure:NoSchedule op=Exists
 ...

Which solution do you suggest?

Make the addition of these default pods.spec.tolerations optional and customizable.

This would be a logical approach since if the initial vCluster pod can be scheduled successfully on a host cluster node (the "pseudo-node"), it implies that it already meets the node's taint requirements without needing these extra tolerations. Therefore, the DaemonSet or debug pods created within that vCluster should not require them either.

Which alternative solutions exist?

For debug pods, a workaround is to get the pod's spec, delete the pod, and then recreate it with the desired (empty) tolerations. However, applying this method to DaemonSet pods is not ideal, as we lose the benefits and control features of the DaemonSet controller.

Additional context

I encountered this missing feature when deploying an instance of the OpenTelemetry (OTEL) Collector CRD in Daemonset mode for a Proof of Concept (POC).

The cluster administrators at our organization have a policy in place that restricts the use of pod.spec.tolerations. I believe this is to simplify cluster maintenance procedures on their end.

[cbron]

Hello @ababila , as far as I am aware vCluster doesn't add any tolerations by default. Though Kubernetes itself does add some, including not-ready. Can you provide your config.yaml ? There is an enforceTolerations setting that may be adding these. Could you also try deploying the operator on a plain cluster (such as kind) to check which tolerations are added ?

[ababila]

Hello @cbron,

This issue isn't necessarily related to the OTEL collector ( I juste initially see it when testing the otel collector with daemonset mode). I can reproduce the same behavior by directly creating a simple DaemonSet, like this one:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: my-daemonset
spec:
  selector:
    matchLabels:
      app: my-daemonset
  template:
    metadata:
      labels:
        app: my-daemonset
    spec:
      containers:
      - name: busybox
        image: busybox
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        command: ["/bin/sh", "-c"]
        args:
          - sleep infinity

The DaemonSet controller automatically adds default tolerations, as documented by Kubernetes. This is the current behavior observed in vCluster (e.g., helm chart 0.25.0 ) and in Kind clusters.

My point is to propose that this behavior be changed within the context of vCluster. Making these default tolerations optional/ configurable for the reasons mentioned above.