origin
origin copied to clipboard
openshift
etcd - All grpc_code for grpc_method "Watch" is "Unavailable"
Hi, I noticed, that every grpc_code for grpc_method "Watch" is "Unavailable" in my okd cluster. My plan is to monitor etcd-instances with default prometheus alerts from the etcd-project. Maybe the watch-connection is not closed correctly and goes into an timeout.
Version
Client Version: 4.7.18
Server Version: 4.7.0-0.okd-2021-08-22-163618
Kubernetes Version: v1.20.0-1093+4593a24e8fd58d-dirty
Steps To Reproduce
- install okd 4.7
- Switch to etcd project
oc project openshift-etcd - Log in to the first etcd member
oc rsh etcd-master1.mycompany.com -
curl -s --cacert "/etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt" --cert "/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master1.mycompany.com.crt" --key "/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-master1.mycompany.com.key" https://localhost:2379/metrics
Current Result
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 1434
Expected Result
grpc_server_handled_total{grpc_code="OK",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 1434
Additional Information
If that behavior is already fixed or it's a false positive, let me know.
Still present with 3.10
oc v3.10.0+0c4577e-1
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://s-cp-lb-01.cloud.example.de:443
openshift v3.10.0+7eee6f8-2
kubernetes v1.10.0+b81c8f8
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
+1 on this. We've disabled this alert on our setup because it's just flapping and not indicating any failures.
+1 on this , I also found it on etcd cluster master node , when add etcd3_alert.rules ..
it will cycle five mintue ... but we can't find something wrong with k8s ....
+1. I run etcd with debug log lever, and find this error:
etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 71; CANCEL")
errors about 1 time ~ in 5 minutes, stream ID - unique
etcd 3.2.24 / 3.2.25 / 3.3.10 Monitoring with prometheus (i getting this allert).
Any updates?
+1, ectd 3.3.10 with Prometheus Operator on Kubernetes 1.11.5
I have 5 nodes, but only one node having the alert, Others seem fine.
the etcd cluster runs well without issue.
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
Relates to https://github.com/openshift/cluster-monitoring-operator/pull/340 and https://github.com/etcd-io/etcd/issues/10289
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
At the moment I am using okd 4.7 and this bug is still present. Prometheus-Query:
grpc_server_handled_total{grpc_code="Unavailable",grpc_service="etcdserverpb.Watch"}