kapacitor
kapacitor copied to clipboard
Alerts suddenly start getting inhibited
This evening, the alert node suddenly started inhibiting alerts even there is no .inhibit() property defined. Events from influxdb continued to be received by kapacitor.
Tick Script Template:
var measurement string
var where_filter = lambda: TRUE
var groups = [*]
var window = 10s
var deadman_interval = 900s
var topic string
var info = lambda: TRUE
var warn = lambda: FALSE
var crit = lambda: FALSE
var data_raw = stream
|from()
.measurement(measurement)
.where(where_filter)
.groupBy(groups)
data_raw
|alert()
.category('threshold')
.info(info)
.warn(warn)
.crit(crit)
.topic(topic)
[...]
Variables:
Vars:
Name Type Value
crit lambda "used_percent" > 90.0
measurement string disk
topic string telegraf_disk
warn lambda "used_percent" > 75.0
Graph:
digraph disk {
graph [throughput="20.01 points/s"];
stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="111222"];
from1 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
from1 -> alert2 [processed="111222"];
alert2 [alerts_inhibited="18148" alerts_triggered="93074" avg_exec_time_ns="0s" crits_triggered="0" errors="0" infos_triggered="92811" oks_triggered="0" warns_triggered="263" working_cardinality="352" ];
Version: kapacitor-1.6.5-1.x86_64 on Centos 7, Influxdb is 1.8.10
Data is gathered from telegraf and sent to influxdb; kapacitor subscribed to the DB. Any pointers?