Alerts when kapacitor is restarted

[thiagocorredor]

Good morning all,

We are facing an issue on kapacitor, related with alerts:

Everytime that the kapacitor service is restarted, we are receiving several INFO, WARNING and CRITICAL alerts.

But these alerts are not necessary, because the previous status were not changed.

Do you know the reason of this?

[reactcker]

Can you provide example: tickscript

[thiagocorredor]

Hi,

This is occurring with all tick scripts.

One example:

var client = 'Client Name'

var measurement = 'lwsn_web_transaction'

stream |from() .measurement(measurement) .groupBy('hostname') |window() .period(1s) .every(0s) |last('inquire_time_taken_status') .as('inquire_time_taken_status') |stateCount(lambda: "inquire_time_taken_status" == 0) .as('status_service_down') |stateCount(lambda: "inquire_time_taken_status" == 1) .as('status_service_up') |eval(lambda: if("inquire_time_taken_status" == 0, 'DOWN', 'UP')) .as('status_service') .keep('inquire_time_taken_status', 'status_service', 'status_service_down', 'status_service_up') |alert() .id('{{ .Name }}') .message('Alert - {{ index .Tags "hostname" }} - Portal Status alert is in a {{ .Level }} state.') .details( '''

Hello!
If you have some question about the alert(s) below, or if you need some assistance as well, please open a ticket

 

Client Name:	''' + client + '''
Hostname:	{{ index .Tags "hostname" }}
Level:	{{ .Level }}
When:	{{ .Time }}
Service Status:	{{ index .Fields "status_service" }}
Details:	{{ index .Tags "inquire_time_taken_error_message" }}

''' ) .info(lambda: "status_service_up" >= 3) .crit(lambda: "status_service_down" >= 3) .stateChangesOnly() .noRecoveries() .topic(measurement) .levelField('level') .IdTag('idtag') .MessageField('message') .email('tsilva@xxxxxx.com') |influxDBOut() .create() .database('xxx') .retentionPolicy('default') .measurement('events') .tag('kapacitor', measurement) .tag('client_name', client)

------ Original Message ------ From: "reactcker" <[email protected]mailto:[email protected]> To: "influxdata/kapacitor" <[email protected]mailto:[email protected]> Cc: "thiagocorredor" <[email protected]mailto:[email protected]>; "Author" <[email protected]mailto:[email protected]> Sent: 13-May-20 10:07:44 AM Subject: Re: [influxdata/kapacitor] Alerts when kapacitor is restarted (#2336)

Can you provide example: tickscript

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/influxdata/kapacitor/issues/2336#issuecomment-627971528, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA77FZCH73NI662VF75JXP3RRKLSBANCNFSM4M62JWDQ.

[thiagocorredor]

Also, is there a reason to kapacitor service restart automatically every night?

[reactcker]

Sorry for the lateness in reply. Hmm it is not odd that kapacitor triggers several INFO, WARNING and CRITICAL alerts every time it meets the condition. Odd is kapacitor getting restart every nigh. Did you look for memory issues might be causing kapacitor to restart?

[thiagocorredor]

Good morning,

I will check about memory issues.

But my question is: Is there a way to avoid kapacitor sending alerts everytime it is restarted?

I mean, a way to save/keep the status of every task, so even restarting kapacitor, it only send alerts if the status change in the future?

Thanks

------ Original Message ------ From: "reactcker" <[email protected]mailto:[email protected]> To: "influxdata/kapacitor" <[email protected]mailto:[email protected]> Cc: "thiagocorredor" <[email protected]mailto:[email protected]>; "Author" <[email protected]mailto:[email protected]> Sent: 18-May-20 5:28:05 AM Subject: Re: [influxdata/kapacitor] Alerts when kapacitor is restarted (#2336)

Sorry for the lateness in reply. Hmm it is not odd that kapacitor triggers several INFO, WARNING and CRITICAL alerts every time it meets the condition. Odd is why it is getting restarted, did you look for memory issues might be causing kapacitor to restart?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/influxdata/kapacitor/issues/2336#issuecomment-630027923, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA77FZD4YKSSZL655TVMZW3RSDWRLANCNFSM4M62JWDQ.

[thiagocorredor]

Actually, this is the real issue:

https://github.com/influxdata/kapacitor/issues/2668