Services Keep Alerting

[spirrello]

Hi,

For some reason every 30 minutes we keep getting alerts on services but no issues have occurred. Is there somewhere to check why? Nothing wrong is showing up in the logs other than successfully posting messages into slack.

We're running K8s 1.54 if it matters.

2017-12-08 16:50:10.306312 I | Message successfully sent to channel G8BBR991D at 1512751810.000104 2017-12-08 16:50:10.469756 I | Message successfully sent to channel G8BBR991D at 1512751810.000194 2017-12-08 16:50:10.585881 I | Message successfully sent to channel G8BBR991D at 1512751810.000261 2017-12-08 16:50:10.697563 I | Message successfully sent to channel G8BBR991D at 1512751810.000325 2017-12-08 16:50:10.804384 I | Message successfully sent to channel G8BBR991D at 1512751810.000406 2017-12-08 16:50:11.022848 I | Message successfully sent to channel G8BBR991D at 1512751810.000550 2017-12-08 16:50:11.126990 I | Message successfully sent to channel G8BBR991D at 1512751810.000625 2017-12-08 16:50:11.239222 I | Message successfully sent to channel G8BBR991D at 1512751810.000687 2017-12-08 16:50:11.367775 I | Message successfully sent to channel G8BBR991D at 1512751810.000761 2017-12-08 16:50:11.483517 I | Message successfully sent to channel G8BBR991D at 1512751811.000077 2017-12-08 16:50:11.604525 I | Message successfully sent to channel G8BBR991D at 1512751811.000159 2017-12-08 16:53:59.056162 I | Message successfully sent to channel G8BBR991D at 1512752038.000661 2017-12-08 16:58:05.028826 I | Message successfully sent to channel G8BBR991D at 1512752284.000578 2017-12-08 17:02:12.410720 I | Message successfully sent to channel G8BBR991D at 1512752531.000804 2017-12-08 17:04:48.564726 I | Message successfully sent to channel G8BBR991D at 1512752688.000195 2017-12-08 17:20:10.395830 I | Message successfully sent to channel G8BBR991D at 1512753610.000134 2017-12-08 17:20:10.505617 I | Message successfully sent to channel G8BBR991D at 1512753610.000191 2017-12-08 17:20:10.597144 I | Message successfully sent to channel G8BBR991D at 1512753610.000241 2017-12-08 17:20:10.686559 I | Message successfully sent to channel G8BBR991D at 1512753610.000293 2017-12-08 17:20:10.765702 I | Message successfully sent to channel G8BBR991D at 1512753610.000354 2017-12-08 17:20:10.905213 I | Message successfully sent to channel G8BBR991D at 1512753610.000408 2017-12-08 17:20:11.043680 I | Message successfully sent to channel G8BBR991D at 1512753610.000492 2017-12-08 17:20:11.139338 I | Message successfully sent to channel G8BBR991D at 1512753610.000535 2017-12-08 17:20:11.230527 I | Message successfully sent to channel G8BBR991D at 1512753611.000040 2017-12-08 17:20:11.351597 I | Message successfully sent to channel G8BBR991D at 1512753611.000110 2017-12-08 17:20:11.451932 I | Message successfully sent to channel G8BBR991D at 1512753611.000167 2017-12-08 17:50:10.326576 I | Message successfully sent to channel G8BBR991D at 1512755409.000657 2017-12-08 17:50:10.422760 I | Message successfully sent to channel G8BBR991D at 1512755409.000719 2017-12-08 17:50:10.738609 I | Message successfully sent to channel G8BBR991D at 1512755410.000161 2017-12-08 17:50:10.830598 I | Message successfully sent to channel G8BBR991D at 1512755410.000220 2017-12-08 17:50:10.959868 I | Message successfully sent to channel G8BBR991D at 1512755410.000305 2017-12-08 17:50:11.109078 I | Message successfully sent to channel G8BBR991D at 1512755410.000387 2017-12-08 17:50:11.212455 I | Message successfully sent to channel G8BBR991D at 1512755410.000460 2017-12-08 17:50:11.336679 I | Message successfully sent to channel G8BBR991D at 1512755411.000066 2017-12-08 17:50:11.433150 I | Message successfully sent to channel G8BBR991D at 1512755411.000107 2017-12-08 17:50:11.535122 I | Message successfully sent to channel G8BBR991D at 1512755411.000170

[ngtuna]

My experience is shouldn't set watch for service. Although the service hasn't changed but I guess kubewatch detects changes in its internal metadata because of incoming requests. I guess in your case it's not delete or add notifies but update, am I right ?

[viveksatasiya]

hi @ngtuna, yes you are right. I am also getting service update notifications on regular intervals. Currently I am removing watch for services.

[ngtuna]

@viveksatasiya 👍 So I guess we can close this

[elvanowen]

Hi @ngtuna, can you share anything or any reasons about the "internal metadata" you mentioned previously ? I am having the same issue here and would still want to listen for any service resource changes when it actually happened.

[codenio]

Hope this is resolved now and we could close this issue.