etcd watchServer goroutine leaks

[fearblackcat]

There are almost 30k goroutines for each breakpoint position as the preceding image. The etcd version tag is on v3.2.7.

And the following is the pprof goroutine logs:

goroutine profile: total 108805
36222 @ 0x42cb1a 0x43bda4 0x43aa0c 0x7a10c6 0x7a1b3c 0x46e7e9 0x46e958 0x7acb29 0x7ad68d 0x7b981d 0xbe767b 0x7de562 0xbf0ffc 0xbf3a7b 0x459d21
#	0x7a10c5	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read+0x555			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/transport.go:143
#	0x7a1b3b	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*Stream).Read+0x5b					/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport/transport.go:333
#	0x46e7e8	io.ReadAtLeast+0xa8													/usr/local/go/src/io/io.go:307
#	0x46e957	io.ReadFull+0x57													/usr/local/go/src/io/io.go:325
#	0x7acb28	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*parser).recvMsg+0x68						/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/rpc_util.go:243
#	0x7ad68c	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.recv+0x4c							/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/rpc_util.go:339
#	0x7b981c	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*serverStream).RecvMsg+0x11c					/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/stream.go:613
#	0xbe767a	github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredServerStream).RecvMsg+0x4a/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:69
#	0x7de561	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.(*watchWatchServer).Recv+0x61		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2514
#	0xbf0ffb	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*serverWatchStream).recvLoop+0x4b	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:176
#	0xbf3a7a	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch.func2+0x2a		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:143

36222 @ 0x42cb1a 0x43bda4 0x43aa0c 0xbf0de9 0x7de447 0xbe7487 0xbf3095 0x7b3f04 0x7b5a65 0x7bb779 0x459d21
#	0xbf0de8	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch+0x4e8	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:147
#	0x7de446	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Watch_Watch_Handler+0xb6	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2495
#	0xbe7486	github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.StreamServerInterceptor+0x106	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:40
#	0xbf3094	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1+0x114	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:93
#	0x7b3f03	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processStreamingRPC+0x353			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:849
#	0x7b5a64	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream+0x12d4				/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:936
#	0x7bb778	github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0xa8			/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:497

36222 @ 0x42cb1a 0x43bda4 0x43aa0c 0xbf2628 0xbf3a2b 0x459d21
#	0xbf2627	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*serverWatchStream).sendLoop+0xb47	/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:298
#	0xbf3a2a	github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch.func1+0x2a		/home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/watch.go:133

Could anyone(boss or dashen) give some help for me?

[wenjiaswe]

@jingyih

[vimalk78]

The Watch api is a streaming api. Its input and output both are streams. The same Watcher, potentially, can be used to watch for multiple keys and ranges.

In the etcdserver, a goroutine handles the invocation of this api

github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch+0x4e8
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Watch_Watch_Handler+0xb6
github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.StreamServerInterceptor+0x106
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1+0x114
github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processStreamingRPC+0x353
github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream+0x12d4
github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0xa8

this goroutine creates 2 new go routines, one for receiving Watch requests on the stream

github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read+0x555
github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/transport.(*Stream).Read+0x5b
io.ReadAtLeast+0xa8
io.ReadFull+0x57
github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*parser).recvMsg+0x68
github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.recv+0x4c
github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*serverStream).RecvMsg+0x11c
github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredServerStream).RecvMsg+0x4a
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.(*watchWatchServer).Recv+0x61
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*serverWatchStream).recvLoop+0x4b
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch.func2+0x2a

and another for sending Watch responses on the response stream.

github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*serverWatchStream).sendLoop+0xb47
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*watchServer).Watch.func1+0x2a	

Ideally the clientv3 can reuse the same streams for one instance of client, and this will create 4 go routines only (3 in server, 1 in client).

the clientv3/watch.go , creates a different stream for different grpc metadata context

@fearblackcat , can you share your client code? do you use single clientv3 instance or multiple? do you use different grpc metadata ?

[fearblackcat]

@vimalk78 I only initialize one instance for clientv3. But the

Endpoints property are all the servers(in my case there are three)

[vimalk78]

@fearblackcat , i tried with the code as in below link https://play.golang.org/p/c-YO-aDTPwC

i didn't observe any increase in go-routines. only single instance of send/recv goroutines is created in serverWatchStream , and is destroyed when all the watches are cancelled. with two instances of above client program, two instances are created and destroyed.

perhaps my client code is simpler than yours.

[jingyih]

@fearblackcat In which test did you observe this issue? I think the first step is for us to reproduce the same issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.