notifications-engine

notifications-engine copied to clipboard

In argocd notifications, I use the following trigger to filter notifications with the oncePer option.

    trigger.on-deployed: |
      - description: Application is synced and healthy. Triggered once per commit.
        when: app.status.health.status == 'Healthy' and app.status.operationState != nil and app.status.operationState.phase in ['Succeeded']
        oncePer: app.status.operationState.syncResult.revision
        send:
        - app-deployed

This returns the following error in the logs

{"level":"error","msg":"failed to execute oncePer condition: cannot fetch syncResult from \u003cnil\u003e (1:27)\n | app.status.operationState.syncResult.revision\n | ..........................^","time":"2022-01-28T15:41:46Z"}

Solution

I think the oncePer condition should not be evaluated when the when condition evaluate to false.

https://github.com/argoproj/notifications-engine/blob/b9d7d6c233cc0c52b9c731f2ce1e909794c4347c/pkg/triggers/service.go#L94

Using the default triggers from the argocd notifications_catalog :

  trigger.on-deployed: |
    - description: Application is synced and healthy. Triggered once per commit.
      oncePer: app.status.operationState.syncResult.revision
      send:
      - app-deployed
      when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'

Ends up yielding the same errors during a sync:

time="2022-04-05T18:43:10Z" level=info msg="Start processing" resource=argocd/blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=debug msg="No OperationState found, SyncStatus is assumed to be up-to-date" app=blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=info msg="Trigger on-deleted result: [{[0].BmlU8P6R blackbox-exporter-gcp [app-deleted] false}]" resource=argocd/blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=error msg="failed to execute when condition: cannot fetch phase from <nil> (1:27)\n | app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy' \n | ..........................^"
time="2022-04-05T18:43:10Z" level=error msg="failed to execute oncePer condition: cannot fetch syncResult from <nil> (1:27)\n | app.status.operationState.syncResult.revision\n | ..........................^"
time="2022-04-05T18:43:10Z" level=info msg="Trigger on-deployed result: [{[0].QeuLK  [app-deployed] false}]" resource=argocd/blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=info msg="Trigger on-health-degraded result: [{[0].zoG5z  [app-health-degraded] false}]" resource=argocd/blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=error msg="failed to execute when condition: cannot fetch phase from <nil> (1:27)\n | app.status.operationState.phase in ['Error', 'Failed']\n | ..........................^"
time="2022-04-05T18:43:10Z" level=info msg="Trigger on-sync-failed result: [{[0].H9Wjs  [app-sync-failed] false}]" resource=argocd/blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=error msg="failed to execute when condition: cannot fetch phase from <nil> (1:27)\n | app.status.operationState.phase in ['Running']\n | ..........................^"
time="2022-04-05T18:43:10Z" level=info msg="Trigger on-sync-running result: [{[0].xGlb  [app-sync-running] false}]" resource=argocd/blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=error msg="failed to execute when condition: cannot fetch phase from <nil> (1:27)\n | app.status.operationState.phase in ['Succeeded']\n | ..........................^"
time="2022-04-05T18:43:10Z" level=info msg="Trigger on-sync-succeeded result: [{[0].zxM90E  [app-sync-succeeded] false}]" resource=argocd/blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=info msg="Trigger on-sync-status-unknown result: [{[0].6SzWb  [app-sync-status-unknown] false}]" resource=argocd/blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=info msg="Trigger on-created result: [{[0].X_5TO4M blackbox-exporter-gcp [app-created] true}]" resource=argocd/blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=info msg="Notification about condition 'on-created.[0].X_5TO4MPCKAyY0ipFgr6_IraRNs' already sent to '{slack general}'" resource=argocd/blackbox-exporter-gcp
time="2022-04-05T18:43:10Z" level=info msg="Processing completed" resource=argocd/blackbox-exporter-gcp

One note, the notifications still work without issues, it's just showing a bunch of error logs. Also should the example argocd notifications_catalog be updated to not use that example?

Seems a consequence of this is that we're spammed with sync is 'Unknown' error-messages in Slack.

Im experiencing something similar:

- description: Application is synced and healthy. Triggered once per commit.
              oncePer: app.status.operationState.syncResult.revision and app.metadata.name
              send:
              - app-deployed
              when: app.status.operationState.phase == 'Succeeded' and app.status.health.status == 'Healthy'

We are using a monorepo for multiple applications and environments. oncePer: app.status.operationState.syncResult.revision and app.metadata.name -> This prevents each application from spamming logs everytime any env or app changes. Its working, but the notification service is spamming logs:

time="2022-11-30T15:05:56Z" level=error msg="failed to execute oncePer condition: cannot fetch revision from <nil> (1:23)
 | app.status.syncResult.revision and app.metadata.name
 | ......................^"

same issue

same issue

I don't use oncePer (unless it's enabled by default) and get the same error:

      trigger.on-sync-failed: |
        - description: Application syncing has failed
          send:
          - app-sync-failed
            when: app.status.operationState.phase in ['Error', 'Failed']
   
      trigger.on-sync-succeeded: |
        - description: Application syncing has succeeded
          send:
          - app-sync-succeeded
            when: app.status.operationState.phase in ['Succeeded']

gives:

notifications-controller {"level":"error","msg":"failed to execute when condition: cannot fetch phase from \u003cnil\u003e (1:27)\n | app.status.operationState.phase in ['Error', 'Failed']\n | ..........................^","time":"20 │
│ 23-02-16T11:40:00Z"}

argocd v2.5.7

We are using oncePer and seeing this on v2.6.2. We had an ApplicationSet that deployed an Application to another cluster that appeared OutOfSync. Once we synced that Application, the messages stopped.

Hitting the same issue

argocd-notifications-controller-7c8dcfd677-72wrh argocd-notifications-controller time="2023-03-10T16:08:06Z" level=error msg="failed to execute when condition: cannot fetch phase from <nil> (1:27)\n | app.status.operationState.phase in ['Error', 'Failed']\n | ..........................^"

  trigger.on-sync-failed: |
    - description: Application syncing has failed
      send:
      - app-sync-failed
      when: app.status.operationState.phase in ['Error', 'Failed']

version: v2.6.4+7be094f

Same thing, notifications working, but log is spammed. version: v2.6.1+3f143c9

Hitting the same issue. Here is our on-sync-failed trigger definition (copied from the official Triggers and Templates catalog):

      trigger.on-sync-failed: |
        - description: Application syncing has failed
          send:
          - app-sync-failed
          when: app.status.operationState.phase in ['Error', 'Failed']

The notification controller send those logs:

failed to execute when condition: cannot fetch phase from <nil> (1:27)
 | app.status.operationState.phase in ['Error', 'Failed']
 | ..........................^

Version: v2.7.2

Same error with defaults triggers from catalog: Version: 2.7.2

notifications-controller logs: cannot fetch phase from <nil> (1:27)\n | app.status.operationState.phase in ['Succeeded']\n | ..........................^"

Experiencing the same behaviour (notifications are working), but different error : time="2023-06-22T09:40:53Z" level=error msg="failed to execute oncePer condition: interface conversion: interface {} is string, not bool (1:47)\n | app.status.operationState.syncResult.revision and app.metadata.name\n | ..............................................^" on v2.7.5

- when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
  oncePer: app.status.operationState.syncResult.revision and app.metadata.name
  send: [slack-deployment-status]

I suggest using the following condition:

oncePer: "[app.status?.operationState?.syncResult?.revision, app.status?.operationState?.syncResult?.revisions]"

This also directly supports multi source apps.

I solved it by syncing the application. app.status.operationState.phase is the data created after sync.

Check whether the app.status.operationState.phase value exists through

kubectl edit application <application>

I suggest using the following condition:

oncePer: "[app.status?.operationState?.syncResult?.revision, app.status?.operationState?.syncResult?.revisions]"

This also directly supports multi source apps.

@woehrl01 how can you added another var to evaluate? Trying to add app.status.operationState.finishedAt like so
oncePer: "[app.status?.sync?.revision, app.status?.sync?.revisions, app.status.operationState.finishedAt]" or oncePer: "[app.status?.sync?.revision, app.status?.sync?.revisions]" and app.status.operationState.finishedAt

neither work

@gespi1 This should work, I guess you are just missing some ? to make status, operationState optional. Otherwise the expression will fail.