etcd icon indicating copy to clipboard operation
etcd copied to clipboard

Published 20 hours ago • verified publisher icon etcd-io

grpc health check crashes etcd

Open maplain opened this issue 3 years ago • 4 comments
trafficstars

What happened?

When a grpc healthcheck is configured for /health, etcd panics with "not implemented"

What did you expect to happen?

etcd handles the health check successfully.

How can we reproduce it (as minimally and precisely as possible)?

Use https://github.com/grpc-ecosystem/grpc-health-probe to do a tls grpc health check on a 3.4.10 server.

Anything else we need to know?

verified 3.5.1 works fine.

Etcd version (please run commands below)

$ etcd --version
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
	A future release of golang/protobuf will delete this package,
	which has long been excluded from the compatibility promise.

etcd Version: 3.4.10
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.14.5
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.4.10
API version: 3.4

Etcd configuration (command line flags or environment variables)

--data-dir=/xx/etcd/data --wal-dir=/xx/etcd/wal --enable-v2=false --enable-grpc-gateway=true --log-outputs=stderr

--quota-backend-bytes=xxx --auto-compaction-retention=1h

--name=xxx --discovery-srv=xxx --discovery-srv-name= --initial-cluster-token=xxx

--peer-client-cert-auth --peer-cert-allowed-hostname=xx --peer-cert-file=/xxx --peer-key-file=/xxx --peer-trusted-ca-file=/xxx --listen-peer-urls=https://10.100.82.202:2380 --initial-advertise-peer-urls=https://xx --listen-metrics-urls=http://10.100.82.202:2381

--client-cert-auth --client-cert-allowed-hostname=xxx --cert-file=/etc/ssl/certs/machine.crt --key-file=/etc/ssl/private/machine.key --trusted-ca-file=/etc/ssl/certs/machine-cas.pem --listen-client-urls=https://10.100.82.202:2379 --advertise-client-urls=https://xxx

--log-level 'info'

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

request:

date && sudo ./grpc_health_probe -tls -tls-ca-cert /etc/ssl/certs/machine-cas.pem -tls-client-cert /etc/ssl/certs/machine.crt -tls-client-key /etc/ssl/private/machine.key -addr=$HOSTNAME:2379
Mon Mar  7 18:24:53 UTC 2022
error: health rpc failed: rpc error: code = Unavailable desc = error reading from server: EOF

etcd 3.4.10 crash log

[2022-03-07 18:24:53.447347] panic: not implemented
[2022-03-07 18:24:53.447383]
[2022-03-07 18:24:53.447397] goroutine 42184 [running]:
[2022-03-07 18:24:53.447415] github.com/golang/protobuf/proto.(*InternalMessageInfo).Unmarshal(...)
[2022-03-07 18:24:53.447432] 	external/com_github_golang_protobuf/proto/deprecated.go:92
[2022-03-07 18:24:53.447456] google.golang.org/grpc/health/grpc_health_v1.(*HealthCheckRequest).XXX_Unmarshal(0xc007802090, 0x0, 0x0, 0x0, 0xc007802090, 0xc000b48a01)
[2022-03-07 18:24:53.447481]  external/org_golang_google_grpc/health/grpc_health_v1/health.pb.go:73 +0x39
[2022-03-07 18:24:53.447522] go.etcd.io/etcd/vendor/github.com/gogo/protobuf/proto.Unmarshal(0x0, 0x0, 0x0, 0x7fe57016fa08, 0xc007802090, 0xa03ac3, 0xc001d6a800)
[2022-03-07 18:24:53.447554]  external/io_etcd_go_etcd/vendor/github.com/gogo/protobuf/proto/decode.go:337 +0x1bc
[2022-03-07 18:24:53.447574] go.etcd.io/etcd/etcdserver/api/v3rpc.(*codec).Unmarshal(0x1b22120, 0x0, 0x0, 0x0, 0x1068580, 0xc007802090, 0x18, 0x30)
[2022-03-07 18:24:53.447597]  external/io_etcd_go_etcd/etcdserver/api/v3rpc/codec.go:29 +0x92
[2022-03-07 18:24:53.447619] google.golang.org/grpc.(*Server).processUnaryRPC.func2(0x1068580, 0xc007802090, 0x30, 0x1057500)
[2022-03-07 18:24:53.447639]  external/org_golang_google_grpc/server.go:1059 +0xd8
[2022-03-07 18:24:53.447665] google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler(0x100c560, 0xc0078035f0, 0x12fdec0, 0xc007802060, 0xc009dc3f20, 0xc007803140, 0x12fdec0, 0xc007802060, 0x0, 0x0)
[2022-03-07 18:24:53.447686]  external/org_golang_google_grpc/health/grpc_health_v1/health.pb.go:289 +0x53
[2022-03-07 18:24:53.447711] google.golang.org/grpc.(*Server).processUnaryRPC(0xc00028b1e0, 0x130dd20, 0xc00a8290e0, 0xc001d6a800, 0xc0078036b0, 0x1ad9dd0, 0x0, 0x0, 0x0)
[2022-03-07 18:24:53.447729]  external/org_golang_google_grpc/server.go:1082 +0x50a
[2022-03-07 18:24:53.447748] google.golang.org/grpc.(*Server).handleStream(0xc00028b1e0, 0x130dd20, 0xc00a8290e0, 0xc001d6a800, 0x0)
[2022-03-07 18:24:53.447768]  external/org_golang_google_grpc/server.go:1405 +0xccd
[2022-03-07 18:24:53.447792] google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00a528a30, 0xc00028b1e0, 0x130dd20, 0xc00a8290e0, 0xc001d6a800)
[2022-03-07 18:24:53.447816]  external/org_golang_google_grpc/server.go:746 +0xa1
[2022-03-07 18:24:53.447836] created by google.golang.org/grpc.(*Server).serveStreams.func1
[2022-03-07 18:24:53.447851]  external/org_golang_google_grpc/server.go:744 +0xa1
[2022-03-07 18:24:53.486004] WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
[2022-03-07 18:24:53.486049]  A future release of golang/protobuf will delete this package,
[2022-03-07 18:24:53.486066]  which has long been excluded from the compatibility promise.
[2022-03-07 18:24:53.486083]
[2022-03-07 18:24:53.486846] [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
[2022-03-07 18:24:53.486883] 2022-03-07 18:24:53.486808 I | etcdmain: etcd Version: 3.4.10
[2022-03-07 18:24:53.486902] 2022-03-07 18:24:53.486832 I | etcdmain: Git SHA: Not provided (use ./build instead of go build)
[2022-03-07 18:24:53.486931] 2022-03-07 18:24:53.486835 I | etcdmain: Go Version: go1.14.5
[2022-03-07 18:24:53.486953] 2022-03-07 18:24:53.486839 I | etcdmain: Go OS/Arch: linux/amd64
[2022-03-07 18:24:53.486980] 2022-03-07 18:24:53.486842 I | etcdmain: setting maximum number of CPUs to 32, total number of available CPUs is 32

On 3.5.1

date && sudo ./grpc_health_probe -tls -tls-ca-cert /etc/ssl/certs/machine-cas.pem -tls-client-cert /etc/ssl/certs/machine.crt -tls-client-key /etc/ssl/private/machine.key -addr=$HOSTNAME:2379
Mon Mar  7 18:29:47 UTC 2022
status: SERVING

etcd --version
etcd Version: 3.5.1
Git SHA: GitNotFound
Go Version: go1.16.2
Go OS/Arch: linux/amd64

etcdctl version
etcdctl version: 3.5.1
API version: 3.5

maplain avatar Mar 07 '22 18:03 maplain

cc @serathius

maplain avatar Mar 17 '22 05:03 maplain

I'm not super familiar with v3.4 release, have you tried to reproduce it on latest release v3.4.18?

serathius avatar Mar 17 '22 10:03 serathius

I'm super familiar with v3.4 release, have you tried to reproduce it on latest release v3.4.18?

is there a fix that goes in v3.4.18?

maplain avatar Mar 24 '22 18:03 maplain

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Jul 10 '22 13:07 stale[bot]