Karpenter keep scheduling pod with PVC on node that reach max EBS volume

[Ph4rell]

Description

Observed Behavior: When a node reach the maximum allocatable volume, Karpenter keep trying to schedule pod with PVC on that node.

Expected Behavior: Karpenter should scale up a new node to schedule new pod with volumes.

Reproduction Steps (Please include YAML):

• Deploy a new nodepool:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: test
spec:
  weight: 10
  template:
    metadata:
      labels:
        role:test
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: firstclass
      startupTaints:
        - key: ebs.csi.aws.com/agent-not-ready
          effect: NoExecute
      taints:
        - key: test
          effect: NoSchedule
      expireAfter: Never
      terminationGracePeriod: 48h
      requirements:
        - key: "karpenter.k8s.aws/instance-family"
          operator: In
          values: ["r4", "r5"]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["xlarge", "2xlarge"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: Never # Added to allow additional control over consolidation aggressiveness
  weight: 10

• Deploy a statefulset that target the node

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test
  labels:
    app: my-app
  namespace: kube-system
spec:
  replicas: 25
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: busybox
        command: ["/bin/sh", "-c", "while true; do sleep1; done"]
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          allowPrivilegeEscalation: false 
          capabilities:
            drop:
              - ALL
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - mountPath: /mnt/data
          name: my-pvc
      tolerations:
      - key: "test"
        operator: "Exists"
        effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: role
                operator: In
                values:
                - test
  volumeClaimTemplates:
    - metadata:
        name: my-pvc
      spec:
        storageClassName: gp2-1a
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 10Gi

• The storageClass look like this:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2-1a
parameters:
  type: gp2
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Versions:

• Karpenter Controller Version: 1.0.6
• Kubernetes Version (kubectl version): v1.30.3
• Server Version: v1.30.4-eks-a737599
• aws-ebs-csi-driver:v1.35.0

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment