[Bug] Second subscriber not triggered and retries not initiated when the first subscriber fails

[kzcPo]

Describe the bug

I have set up a series of subscribers that should automatically switch to the next one upon failure of the previous, along with an expected retry mechanism under certain conditions.

The configuration is as follows：

subscriptions: |
    # subscription for on-sync-status-unknown trigger notifications
    - recipients:
      - feishu
      triggers:
      - on-rollout-completed
    - recipients:
      - jenkins
      triggers:
      - on-rollout-completed

In practice, however, when the first-subscriber encounters a failure, it does not seem to trigger the execution of the second-subscriber, nor does it initiate any retries as configured. Instead, the process seems to terminate prematurely.

Version

argo-rollouts:v1.6.0

Logs

time="2024-02-01T03:26:18Z" level=info msg="Start processing" resource=gray/message
time="2024-02-01T03:26:18Z" level=info msg="Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-completed] false}]" resource=gray/message
time="2024-02-01T03:26:18Z" level=info msg="Processing completed" resource=gray/message
time="2024-02-01T03:26:18Z" level=error msg="notification error: request to {{\n    \"msg_type\": \"interactive\",\n    \"card\": {\n      \"config\": {\n        \"wide_screen_mode\": true\n      },\n      \"elements\": [\n        {\n          \"tag\": \"markdown\",\n          \"content\": \"**应用**：message\\n**环境**：gray\\n**状态**：Paused\\n**镜像**：[**fpm**]xxx.com/xxx/message:online_social_release_v1\"\n        }\n      ],\n      \"header\": {\n        \"template\": \"blue\",\n        \"title\": {\n          \"content\": \"message 更新通知 Rollout completed\",\n          \"tag\": \"plain_text\"\n        }\n      }\n    }\n}\n POST https://open.feishu.cn/open-apis/bot/v2/hook/xxxx feishu} has failed with error code 400 : {\"code\":9499,\"msg\":\"too many request\",\"data\":{}}"
time="2024-02-01T03:26:18Z" level=error msg="Notifications failed to send for eventReason RolloutCompleted with error: request to {{\n    \"msg_type\": \"interactive\",\n    \"card\": {\n      \"config\": {\n        \"wide_screen_mode\": true\n      },\n      \"elements\": [\n        {\n          \"tag\": \"markdown\",\n          \"content\": \"**应用**：message\\n**环境**：gray\\n**状态**：Paused\\n**镜
像**：[**fpm**]xxx.com/xxx/message:online_social_release_v1\"\n        }\n      ],\n      \"header\": {\n        \"template\": \"blue\",\n        \"title\": {\n          \"content\": \"message 更新通知 Rollout completed\",\n          \"tag\": \"plain_text\"\n        }\n      }\n    }\n}\n POST https://open.feishu.cn/open-apis/bot/v2/hook/xxxx feishu} has failed with error code 400 : {\"code\":9499,\"msg\":\"too many request\",\"data\":{}}" event_reason=RolloutCompleted namespace=gray rollout=message
time="2024-02-01T03:26:18Z" level=info msg="Rollout completed update to revision 62 (585f786d7d): Completed blue-green update" event_reason=RolloutCompleted namespace=gray rollout=message