Trigger does not send the alert if trigger condition is set to less than 0 and value of monitoring metric is negative

[kklmm]

Hello,

problem: When setting up monitor/trigger alerting on negative values no alert is being sent even though the monitoring value is way below 0 (negative). expected result: send an alert if a value is below 0.

Details:

I've set up a monitor using visual graph that checks the min() of a value for the past hour. See: https://i.imgur.com/tKfOg50.png

And a trigger to send an alert on slack if the monitoring value is less than -1. You can see on the chart that the value is way below the threshold (from -30 to -40 on the example chart) and alert should be sent. However, no alert is sent, ever for any time period. https://i.imgur.com/evbcsq7.png

Alert history is green all the way. https://i.imgur.com/PibHc4n.png Should be red and triggered.

Testing the alert with "send test message" under configure actions works fine and the test alert is indeed sent so it's not a delivery issue.

[dbbaughe]

Hi @kklemen,

I tested a monitor and was able to receive an alert when using min aggregation and negative numbers. Can you post the JSON of your monitor (from the GET monitor/{id} API) so I can take a look (don't forget to scrub any data from the JSON!).

Thanks, Drew

[kklmm]

Hi Drew, thanks for the quick response.

Here's the json:

{
  "_id" : "Y5VSrmsBFHcyHWAZO56m",
  "_version" : 6,
  "_seq_no" : 23,
  "_primary_term" : 4,
  "monitor" : {
    "type" : "monitor",
    "schema_version" : 1,
    "name" : "stats-pnrque alert if queue",
    "enabled" : true,
    "enabled_time" : 1561997753250,
    "schedule" : {
      "period" : {
        "interval" : 1,
        "unit" : "HOURS"
      }
    },
    "inputs" : [
      {
        "search" : {
          "indices" : [
            "stats-pnrqueue*"
          ],
          "query" : {
            "size" : 0,
            "query" : {
              "bool" : {
                "filter" : [
                  {
                    "range" : {
                      "@timestamp" : {
                        "from" : "{{period_end}}||-1h",
                        "to" : "{{period_end}}",
                        "include_lower" : true,
                        "include_upper" : true,
                        "format" : "epoch_millis",
                        "boost" : 1.0
                      }
                    }
                  }
                ],
                "adjust_pure_negative" : true,
                "boost" : 1.0
              }
            },
            "aggregations" : {
              "when" : {
                "min" : {
                  "field" : "next_exec_time_in_hours"
                }
              }
            }
          }
        }
      }
    ],
    "triggers" : [
      {
        "id" : "qpVUrmsBFHcyHWAZ9apf",
        "name" : "stats-pnrqueue alert if queue",
        "severity" : "1",
        "condition" : {
          "script" : {
            "source" : "return ctx.results[0].aggregations.when.value == null ? false : ctx.results[0].aggregations.when.value < -1",
            "lang" : "painless"
          }
        },
        "actions" : [
          {
            "id" : "q5VUrmsBFHcyHWAZ9apf",
            "name" : "alert if stats-pnrqueue queue",
            "destination_id" : "b1f8mWsBKhqornm_gLHj",
            "message_template" : {
              "source" : """
Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.
https://linktokibanadashboard?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now))

- Trigger: {{ctx.trigger.name}}
- Severity: {{ctx.trigger.severity}}
- Period start: {{ctx.periodStart}}
- Period end: {{ctx.periodEnd}}
""",
              "lang" : "mustache"
            },
            "throttle_enabled" : false,
            "subject_template" : {
              "source" : "Negative queue in stats-pnrqueue",
              "lang" : "mustache"
            }
          }
        ]
      }
    ],
    "last_update_time" : 1563612876058
  }
}

[lezzago]

Hi @kklemen,

I went through your monitor information and it appears there is nothing wrong with its data. I also tested a monitor and was able to receive an alert when using min aggregation and negative numbers, so I could not reproduce your error.

Are you still receiving this problem? If you still are, it would be great to know which ODFE version you are using. Also this issue should not appear if you are willing to upgrade to the current release.

Thanks, Ashish