Provide a way of knowing that a cluster is scaling up or down

[johnstairs]

Maybe the most straightforward way is to expose data from the cluster autoscaler:

kubectl get configmap -n kube-system cluster-autoscaler-status -o yaml                                                                         c97b3889a26b: Wed Nov 29 13:34:30 2023

apiVersion: v1
data:
  status: |+
    Cluster-autoscaler status at 2023-11-29 13:34:29.001925665 +0000 UTC:
    Cluster-wide:
      Health:      Healthy (ready=1 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=1 longUnregistered=0)
                   LastProbeTime:      2023-11-29 13:34:28.808043334 +0000 UTC m=+1081389.237180180
                   LastTransitionTime: 2023-11-21 22:52:15.452499092 +0000 UTC m=+423655.881636038
      ScaleUp:     InProgress (ready=1 registered=1)
                   LastProbeTime:      2023-11-29 13:34:28.808043334 +0000 UTC m=+1081389.237180180
                   LastTransitionTime: 2023-11-29 13:34:01.090096765 +0000 UTC m=+1081361.519233711
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2023-11-29 13:34:28.808043334 +0000 UTC m=+1081389.237180180
                   LastTransitionTime: 2023-11-28 20:53:08.193145649 +0000 UTC m=+1021308.622282595

    NodeGroups:
      Name:        aks-cpunp-38371103-vmss
      Health:      Healthy (ready=0 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=0 longUnregistered=0 cloudProviderTarget=1 (minSize=0, maxSize=10))
                   LastProbeTime:      0001-01-01 00:00:00 +0000 UTC
                   LastTransitionTime: 2023-11-21 22:54:04.887552965 +0000 UTC m=+423765.316689911
      ScaleUp:     InProgress (ready=0 cloudProviderTarget=1)
                   LastProbeTime:      0001-01-01 00:00:00 +0000 UTC
                   LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2023-11-29 13:34:28.808043334 +0000 UTC m=+1081389.237180180
                   LastTransitionTime: 2023-11-28 20:53:08.193145649 +0000 UTC m=+1021308.622282595

      Name:        aks-gpunp-38371103-vmss
      Health:      Healthy (ready=0 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=0 longUnregistered=0 cloudProviderTarget=1 (minSize=0, maxSize=10))
                   LastProbeTime:      0001-01-01 00:00:00 +0000 UTC
                   LastTransitionTime: 2023-11-22 03:40:23.754896837 +0000 UTC m=+440944.184033683
      ScaleUp:     InProgress (ready=0 cloudProviderTarget=1)
                   LastProbeTime:      0001-01-01 00:00:00 +0000 UTC
                   LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2023-11-29 13:34:28.808043334 +0000 UTC m=+1081389.237180180
                   LastTransitionTime: 2023-11-28 20:50:26.717238142 +0000 UTC m=+1021147.146375188

      Name:        aks-system-38371103-vmss
      Health:      Healthy (ready=1 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=1 longUnregistered=0 cloudProviderTarget=1 (minSize=1, maxSize=3))
                   LastProbeTime:      2023-11-29 13:34:28.808043334 +0000 UTC m=+1081389.237180180
                   LastTransitionTime: 2023-11-21 22:52:15.452499092 +0000 UTC m=+423655.881636038
      ScaleUp:     NoActivity (ready=1 cloudProviderTarget=1)
                   LastProbeTime:      2023-11-29 13:34:28.808043334 +0000 UTC m=+1081389.237180180
                   LastTransitionTime: 2023-11-21 22:52:15.452499092 +0000 UTC m=+423655.881636038
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2023-11-29 13:34:28.808043334 +0000 UTC m=+1081389.237180180
                   LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC

kind: ConfigMap
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/last-updated: 2023-11-29 13:34:29.001925665 +0000
      UTC
  creationTimestamp: "2023-11-21T22:52:24Z"
  name: cluster-autoscaler-status
  namespace: kube-system
  resourceVersion: "19489106"
  uid: df89f85d-9bdd-46f5-8678-34db37f80b6d