Graceful scaling of nodes with daemonset.

[derklauer]

What is the recommended method to handle scaling of nodes to prevent node storms when an app has a dependency of node state label sys-box-runtime=running . The initialization time of of sys-box causes conflicts with autos scaling groups. Current configuration:

Karpeneter scaling

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: foo
spec:
  template:
    metadata:
      labels:
        karpenter.sh/capacity-type: on-demand
    spec:
      startupTaints:
      - key: karpenter.sh/wait
        value: "true"
        effect: NoSchedule
  limits:
    cpu: 1000

Testing this configuration but sys-box-runtime=Initialization is slow and causes node storms as it waits for sys-box-runtime=running

apiVersion: apps/v1
kind: DaemonSet
metadata:
  ...
spec:
  ...
    spec:
      tolerations:
        - key: karpenter.sh/wait
          operator: Exists
          effect: NoSchedule
     ...

I would prefer not to solve this by throwing higher compute specs at it and can wait for the daemon set to be in a running state when nodes scale. Is there a better method?