karpenter-provider-aws icon indicating copy to clipboard operation
karpenter-provider-aws copied to clipboard

kubernetes.io/hostname label not working in NodeAffinity for Aerospike Kubernetes Operator

Open abhishekdwivedi3060 opened this issue 6 months ago • 1 comments

Similar issue in Karpenter: https://github.com/aws/karpenter-provider-aws/issues/4671 Related issue in Aerospike: https://github.com/aerospike/aerospike-kubernetes-operator/issues/305

Use-case: There is a feature in Aerospike Kubernetes Operator (AKO) called k8sNodeBlockList (list of K8s node names) where a user can define a list of K8s nodes that should be ignored from scheduling for Aerospike Cluster pods. This feature of AKO helps users in K8s cluster maintenance by migrating pods to other K8s nodes. It uses kubernetes.io/hostname label along with NotIn operator in the NodeAffinity to move pods away from those nodes.

        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: NotIn
                  values:
                  - gke-abhisek-test-default-pool-d04arw3-r5ts

Issue: Karpenter has a sweeping check where it blocks the kubernetes.io/hostname in NodeAffinity. Ref code: https://github.com/kubernetes-sigs/karpenter/blob/d5660acf4472db796d5f4fac58a147d14b320451/pkg/apis/v1beta1/labels.go#L90 As a result if there are pending pods with kubernetes.io/hostname NodeAffinity, they remain in pending state as Karpenter doesn't scale K8s node.

Questions:

  1. Is there a plan to remove that sweeping check for kubernetes.io/hostname label?
  2. Is it possible to only block In operator and allow NotIn operator for kubernetes.io/hostname label?
  3. Is there a work-around possible to bypass that check?

abhishekdwivedi3060 avatar Aug 22 '24 14:08 abhishekdwivedi3060