kapacitor icon indicating copy to clipboard operation
kapacitor copied to clipboard

Alerts suddenly start getting inhibited

Open bbczeuz opened this issue 2 years ago • 0 comments

This evening, the alert node suddenly started inhibiting alerts even there is no .inhibit() property defined. Events from influxdb continued to be received by kapacitor.

Tick Script Template:

var measurement string

var where_filter = lambda: TRUE

var groups = [*]

var window = 10s

var deadman_interval = 900s

var topic string

var info = lambda: TRUE

var warn = lambda: FALSE

var crit = lambda: FALSE

var data_raw = stream
    |from()
        .measurement(measurement)
        .where(where_filter)
        .groupBy(groups)

data_raw
    |alert()
        .category('threshold')
        .info(info)
        .warn(warn)
        .crit(crit)
        .topic(topic)
[...]

Variables:

Vars:
Name                          Type      Value                                   
crit                          lambda    "used_percent" > 90.0                   
measurement                   string    disk                                    
topic                         string    telegraf_disk                           
warn                          lambda    "used_percent" > 75.0                   

Graph:

digraph disk {
graph [throughput="20.01 points/s"];

stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="111222"];

from1 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
from1 -> alert2 [processed="111222"];

alert2 [alerts_inhibited="18148" alerts_triggered="93074" avg_exec_time_ns="0s" crits_triggered="0" errors="0" infos_triggered="92811" oks_triggered="0" warns_triggered="263" working_cardinality="352" ];

Version: kapacitor-1.6.5-1.x86_64 on Centos 7, Influxdb is 1.8.10

Data is gathered from telegraf and sent to influxdb; kapacitor subscribed to the DB. Any pointers?

bbczeuz avatar Jun 21 '23 18:06 bbczeuz