kubewatch icon indicating copy to clipboard operation
kubewatch copied to clipboard
verified publisher icon vmware-archive

Ignore some spurious updates

Open Aracki opened this issue 6 years ago • 2 comments

I'm getting the following message every few minutes.

A daemonset in namespace  has been updated: kube-system/weave-net

What can cause this to happen so often?

Aracki avatar Jan 03 '20 14:01 Aracki

I don't know; can you run kubectl get -o yaml periodically and diff?

perhaps kubewatch should be configurable in such a way that some common uninteresting update causes can be ignored?

mkmik avatar Jul 02 '20 15:07 mkmik

@mkmik I diffed before and after status of the meaningless message. The changed attributes are these; LastProbeTime, cluster-autoscaler.kubernetes.io/last-updated, resourceVersion. Also added raw log. Could you guys ignore these? The logging rates are brutal to use. Screen Shot 2020-11-18 at 4 20 16 PM

before:

apiVersion: v1
data:
  status: |+
    Cluster-autoscaler status at 2020-11-18 07:15:56.309155103 +0000 UTC:
    Cluster-wide:
      Health:      Healthy (ready=19 unready=0 notStarted=0 longNotStarted=0 registered=19 longUnregistered=0)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleUp:     NoActivity (ready=19 registered=19)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:32:36.872155866 +0000 UTC m=+1281.835033375
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:32:46.892169217 +0000 UTC m=+1291.855046784

    NodeGroups:
      Name:        eks-c4bad894-5f38-017e-b619-1ee30c778295
      Health:      Healthy (ready=2 unready=0 notStarted=0 longNotStarted=0 registered=2 longUnregistered=0 cloudProviderTarget=2 (minSize=2, maxSize=2))
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleUp:     NoActivity (ready=2 cloudProviderTarget=2)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906

      Name:        eks-e2badaec-62c0-feb8-dc38-e1c1d348ad74
      Health:      Healthy (ready=7 unready=0 notStarted=0 longNotStarted=0 registered=7 longUnregistered=0 cloudProviderTarget=7 (minSize=1, maxSize=50))
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleUp:     NoActivity (ready=7 cloudProviderTarget=7)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:28:25.316977267 +0000 UTC m=+1030.279854934

      Name:        eksctl-cluster-spot-live-1-nodegroup-ng-4vcpu-16gb-spot-NodeGroup-1MM1WKM1OWW9I
      Health:      Healthy (ready=7 unready=0 notStarted=0 longNotStarted=0 registered=7 longUnregistered=0 cloudProviderTarget=7 (minSize=1, maxSize=30))
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleUp:     NoActivity (ready=7 cloudProviderTarget=7)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:32:36.872155866 +0000 UTC m=+1281.835033375
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:32:46.892169217 +0000 UTC m=+1291.855046784

      Name:        eksctl-cluster-spot-live-1-nodegroup-ng-8vcpu-32gb-spot-NodeGroup-1GZN9B9KMB8L6
      Health:      Healthy (ready=3 unready=0 notStarted=0 longNotStarted=0 registered=3 longUnregistered=0 cloudProviderTarget=3 (minSize=1, maxSize=30))
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:17:38.747012606 +0000 UTC m=+383.709890101
      ScaleUp:     NoActivity (ready=3 cloudProviderTarget=3)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:17:38.747012606 +0000 UTC m=+383.709890101
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2020-11-18 07:15:56.305252488 +0000 UTC m=+11081.268129995
                   LastTransitionTime: 2020-11-18 04:22:51.024282597 +0000 UTC m=+695.987160114

kind: ConfigMap
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/last-updated: 2020-11-18 07:15:56.309155103 +0000
      UTC
  creationTimestamp: "2020-11-17T09:37:33Z"
  name: cluster-autoscaler-status
  namespace: kube-system
  resourceVersion: "4035734"
  selfLink: /api/v1/namespaces/kube-system/configmaps/cluster-autoscaler-status
  uid: 6fb0fd82-51af-45e8-8d5b-5c473a8b7dcc

after:

apiVersion: v1
data:
  status: |+
    Cluster-autoscaler status at 2020-11-18 07:16:06.330970491 +0000 UTC:
    Cluster-wide:
      Health:      Healthy (ready=19 unready=0 notStarted=0 longNotStarted=0 registered=19 longUnregistered=0)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleUp:     NoActivity (ready=19 registered=19)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:32:36.872155866 +0000 UTC m=+1281.835033375
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:32:46.892169217 +0000 UTC m=+1291.855046784

    NodeGroups:
      Name:        eks-c4bad894-5f38-017e-b619-1ee30c778295
      Health:      Healthy (ready=2 unready=0 notStarted=0 longNotStarted=0 registered=2 longUnregistered=0 cloudProviderTarget=2 (minSize=2, maxSize=2))
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleUp:     NoActivity (ready=2 cloudProviderTarget=2)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906

      Name:        eks-e2badaec-62c0-feb8-dc38-e1c1d348ad74
      Health:      Healthy (ready=7 unready=0 notStarted=0 longNotStarted=0 registered=7 longUnregistered=0 cloudProviderTarget=7 (minSize=1, maxSize=50))
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleUp:     NoActivity (ready=7 cloudProviderTarget=7)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:28:25.316977267 +0000 UTC m=+1030.279854934

      Name:        eksctl-cluster-spot-live-1-nodegroup-ng-4vcpu-16gb-spot-NodeGroup-1MM1WKM1OWW9I
      Health:      Healthy (ready=7 unready=0 notStarted=0 longNotStarted=0 registered=7 longUnregistered=0 cloudProviderTarget=7 (minSize=1, maxSize=30))
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:11:56.752906306 +0000 UTC m=+41.715783906
      ScaleUp:     NoActivity (ready=7 cloudProviderTarget=7)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:32:36.872155866 +0000 UTC m=+1281.835033375
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:32:46.892169217 +0000 UTC m=+1291.855046784

      Name:        eksctl-cluster-spot-live-1-nodegroup-ng-8vcpu-32gb-spot-NodeGroup-1GZN9B9KMB8L6
      Health:      Healthy (ready=3 unready=0 notStarted=0 longNotStarted=0 registered=3 longUnregistered=0 cloudProviderTarget=3 (minSize=1, maxSize=30))
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:17:38.747012606 +0000 UTC m=+383.709890101
      ScaleUp:     NoActivity (ready=3 cloudProviderTarget=3)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:17:38.747012606 +0000 UTC m=+383.709890101
      ScaleDown:   NoCandidates (candidates=0)
                   LastProbeTime:      2020-11-18 07:16:06.325736162 +0000 UTC m=+11091.288613715
                   LastTransitionTime: 2020-11-18 04:22:51.024282597 +0000 UTC m=+695.987160114

kind: ConfigMap
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/last-updated: 2020-11-18 07:16:06.330970491 +0000
      UTC
  creationTimestamp: "2020-11-17T09:37:33Z"
  name: cluster-autoscaler-status
  namespace: kube-system
  resourceVersion: "4035791"
  selfLink: /api/v1/namespaces/kube-system/configmaps/cluster-autoscaler-status
  uid: 6fb0fd82-51af-45e8-8d5b-5c473a8b7dcc

dable-mj111 avatar Nov 18 '20 07:11 dable-mj111