argo-kube-notifier icon indicating copy to clipboard operation
argo-kube-notifier copied to clipboard

Unable to monitor container Restarts

Open bjmask opened this issue 4 years ago • 5 comments

Describe the bug

No alerts triggered when restartCount > 0.

To Reproduce apiVersion: argoproj.io/v1alpha1 kind: Notification metadata: name: "dummy-notification" namespace: dev spec: Namespace: dev monitorResource: Resource: pods Version: v1 rules: - allConditions: - jsonPath: status/containerStatuses[0]/restartCount operator: gt value: "0" events: - message: "Condition Triggered : Pod ={{.metadata.name}} is being tested" emailSubject: "[ALERT] Argo Notification Condition Triggered {{.metadata.name}}" notificationLevel: info notifierNames: - slack name: "dummy-notification" initialDelaySec: 1 throttleMinutes: 1 notifiers: - name: slack slack: channel: alerts hookUrlSecret: key: hookURL name: my-slack-secret

Expected behavior

I expect a notification everytime a pod has had a restart greater than 0.

  • Latest image built locally from master.

bjmask avatar May 14 '20 16:05 bjmask

This might work? - jsonPath: status/containerStatuses/*/restartCount

tiej-dr avatar May 15 '20 06:05 tiej-dr

That does indeed work, and it will monitor all containers under a pod. But how can I extract the restart count for the ith container?

e.g templating the restartCount in the message.

{{.status.containerStatuses.[0].restartCount}}

bjmask avatar May 15 '20 18:05 bjmask

Had similar problem addressing list elements - the syntax is not that intuitive. Try something like this: {{ (index .status.containerStatuses 0).restartCount }}

Or more fail-safe, check all containers:

message: |
Pod: {{.metadata.name}}
{{- range $index, $container := .status.containerStatuses }}
  {{- if gt $container.restartCount <threshold> }}
  Container {{$container.containerID}} restarted {{ $container.restartCount }} times. 
  {{ end -}}
{{ end -}}

tiej-dr avatar May 16 '20 12:05 tiej-dr

Should this be added to the README?

bjmask avatar May 19 '20 15:05 bjmask

I'm not involved in this project, so can't give you an answer to that one :)

tiej-dr avatar May 19 '20 19:05 tiej-dr