Add logging grpc request and response content with grpc-proxy mode

[stefanmonkey]

In our test environment, it may be very useful to debug who delete etcd's key with grpc-proxy mode inspired by https://github.com/grpc-ecosystem/go-grpc-middleware

Signed-off-by: stefanbo [email protected]

[ahrtr]

Two comments:

1. Please add a changelog item;
2. Please provide your test result/examples.

[stefanmonkey]

Environment

etcd server: 192.168.56.11:23790 in vagrant ubuntu etcd grpc-proxy: localhost:2379 in macos

Start scripts

• etcd server

export REGISTRY=quay.io/coreos/etcd

docker run -d \
  -p 23790:23790 \
  -p 23800:23800 \
  --name etcd ${REGISTRY}:latest \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name node1 \
  --initial-advertise-peer-urls http://0.0.0.0:23800 --listen-peer-urls http://0.0.0.0:23800 \
  --advertise-client-urls http://0.0.0.0:23790 --listen-client-urls http://0.0.0.0:23790 \
  --initial-cluster node1=http://0.0.0.0:23800

• etcd grpc-proxy

./etcd grpc-proxy start \
--debug \
--experimental-enable-grpc-logging=true \
--endpoints=192.168.56.11:23790 \
--listen-addr=127.0.0.1:2379

Test process

./etcdctl put greeting "Hello, world"
# logging content
{"level":"info","ts":"2022-07-29T10:14:41.890+0800","caller":"zap/payload_interceptors.go:131","msg":"server request payload logged as grpc.request.content field","system":"grpc","span.kind":"server","grpc.service":"etcdserverpb.KV","grpc.method":"Put","peer.address":"127.0.0.1:54276","grpc.request.content":{"msg":{"key":"Z3JlZXRpbmc=","value":"SGVsbG8sIHdvcmxk"}}}
{"level":"info","ts":"2022-07-29T10:14:41.891+0800","caller":"zap/payload_interceptors.go:131","msg":"server response payload logged as grpc.response.content field","system":"grpc","span.kind":"server","grpc.service":"etcdserverpb.KV","grpc.method":"Put","peer.address":"127.0.0.1:54276","grpc.response.content":{"msg":{"header":{"clusterId":"12526192129151561082","memberId":"3877455311180318034","revision":"2","raftTerm":"2"}}}}

./etcdctl get greeting
# logging content
{"level":"info","ts":"2022-07-29T10:16:24.250+0800","caller":"zap/payload_interceptors.go:131","msg":"server request payload logged as grpc.request.content field","system":"grpc","span.kind":"server","grpc.service":"etcdserverpb.KV","grpc.method":"Range","peer.address":"127.0.0.1:54297","grpc.request.content":{"msg":{"key":"Z3JlZXRpbmc="}}}
{"level":"info","ts":"2022-07-29T10:16:24.251+0800","caller":"zap/payload_interceptors.go:131","msg":"server response payload logged as grpc.response.content field","system":"grpc","span.kind":"server","grpc.service":"etcdserverpb.KV","grpc.method":"Range","peer.address":"127.0.0.1:54297","grpc.response.content":{"msg":{"header":{"clusterId":"12526192129151561082","memberId":"3877455311180318034","revision":"2","raftTerm":"2"},"kvs":[{"key":"Z3JlZXRpbmc=","createRevision":"2","modRevision":"2","version":"1","value":"SGVsbG8sIHdvcmxk"}],"count":"1"}}}

summary

In Test deployment, with grpc-proxy before etcdserver, --experimental-enable-grpc-logging=true will help a lot to debug

[ahrtr]

All comments resolved, please @spzala take another look, thx