karpenter-provider-aws
karpenter-provider-aws copied to clipboard
kubernetes.io/hostname label not working in NodeAffinity for Aerospike Kubernetes Operator
Similar issue in Karpenter: https://github.com/aws/karpenter-provider-aws/issues/4671 Related issue in Aerospike: https://github.com/aerospike/aerospike-kubernetes-operator/issues/305
Use-case: There is a feature in Aerospike Kubernetes Operator (AKO) called k8sNodeBlockList (list of K8s node names) where a user can define a list of K8s nodes that should be ignored from scheduling for Aerospike Cluster pods. This feature of AKO helps users in K8s cluster maintenance by migrating pods to other K8s nodes. It uses kubernetes.io/hostname
label along with NotIn
operator in the NodeAffinity to move pods away from those nodes.
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: NotIn
values:
- gke-abhisek-test-default-pool-d04arw3-r5ts
Issue: Karpenter has a sweeping check where it blocks the kubernetes.io/hostname
in NodeAffinity. Ref code: https://github.com/kubernetes-sigs/karpenter/blob/d5660acf4472db796d5f4fac58a147d14b320451/pkg/apis/v1beta1/labels.go#L90
As a result if there are pending pods with kubernetes.io/hostname
NodeAffinity, they remain in pending state as Karpenter doesn't scale K8s node.
Questions:
- Is there a plan to remove that sweeping check for
kubernetes.io/hostname
label? - Is it possible to only block
In
operator and allowNotIn
operator forkubernetes.io/hostname
label? - Is there a work-around possible to bypass that check?