milvus [Bug]: milvus standalone 2.3.2 panic

[Bug]: milvus standalone 2.3.2 panic

Open zzggfly opened this issue 1 year ago • 10 comments

trafficstars

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version: 2.3.2
- Deployment mode(standalone or cluster): standalone 
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): golang v2
- OS(Ubuntu or CentOS):  CentOS
- CPU/Memory: 
- GPU: none
- Others:

Current Behavior

the docker is running, but cannot connect

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

Uploading milvus.log…

Anything else?

No response

Dec 26 '23 08:12 zzggfly

2023/12/26 07:50:38 maxprocs: Updating GOMAXPROCS=8: determined from CPU quota

__  _________ _   ____  ______

/ |/ / / /| | / / / / / __/
/ /|/ // // /| |/ / // /\ \
// /////_/__/

Welcome to use Milvus! Version: v2.3.2-dev Built: Thu Oct 26 09:30:40 UTC 2023 GitCommit: 8b453454d GoVersion: go version go1.20.7 linux/amd64

open pid file: /run/milvus/standalone.pid lock pid file: /run/milvus/standalone.pid [2023/12/26 07:50:38.285 +00:00] [INFO] [roles/roles.go:304] ["starting running Milvus components"] [2023/12/26 07:50:38.285 +00:00] [INFO] [roles/roles.go:167] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so] [2023/12/26 07:50:38.291 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=FileSource] [2023/12/26 07:50:38.291 +00:00] [DEBUG] [config/etcd_source.go:49] ["init etcd source"] [etcdInfo="{"UseEmbed":false,"UseSSL":false,"Endpoints":["cess-etcd-service:2379"],"KeyPrefix":"by-dev-milvus","CertFile":"/path/to/etcd-client.pem","KeyFile":"/path/to/etcd-client-key.pem","CaCertFile":"/path/to/ca.pem","MinVersion":"1.3","RefreshInterval":5000000000}"] [2023/12/26 07:50:38.292 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[cess-etcd-service:2379]"] [minVersion=1.3] [2023/12/26 07:50:38.292 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev-milvus/config] [endpoints="[cess-etcd-service:2379]"] [2023/12/26 07:50:38.297 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=EtcdSource] [2023/12/26 07:50:38.299 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=FileSource] [2023/12/26 07:50:38.299 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}] [2023/12/26 07:50:38.299 +00:00] [INFO] [roles/roles.go:253] [setupPrometheusHTTPServer] [2023/12/26 07:50:38.299 +00:00] [INFO] [http/server.go:77] ["management listen"] [addr=:9091] [2023/12/26 07:50:38.299 +00:00] [INFO] [rootcoord/root_coord.go:155] ["update rootcoord state"] [state=Abnormal] [2023/12/26 07:50:38.299 +00:00] [DEBUG] [rootcoord/service.go:166] ["init params done.."] [2023/12/26 07:50:38.299 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[cess-etcd-service:2379]"] [minVersion=1.3] [2023/12/26 07:50:38.299 +00:00] [INFO] [proxy/lb_policy.go:78] ["use look_aside policy on replica selection"] [2023/12/26 07:50:38.300 +00:00] [DEBUG] [rootcoord/service.go:183] ["etcd connect done ..."] [2023/12/26 07:50:38.300 +00:00] [DEBUG] [rootcoord/service.go:243] ["start grpc "] [port=53100] [2023/12/26 07:50:38.300 +00:00] [INFO] [runtime/asm_amd64.s:1598] ["Start check query node health loop"] [2023/12/26 07:50:38.300 +00:00] [DEBUG] [proxy/multi_rate_limiter.go:188] ["RateLimiter register for rateType"] [rateType=DQLSearch] [rateLimit=+inf] [burst=1.7976931348623157e+308] [2023/12/26 07:50:38.300 +00:00] [DEBUG] [proxy/multi_rate_limiter.go:188] ["RateLimiter register for rateType"] [rateType=DMLUpsert] [rateLimit=+inf] [burst=1.7976931348623157e+308] [2023/12/26 07:50:38.300 +00:00] [DEBUG] [proxy/service.go:113] ["create a new Proxy instance"] [state=2] [2023/12/26 07:50:38.300 +00:00] [DEBUG] [proxy/service.go:365] ["init Proxy server"] [2023/12/26 07:50:38.300 +00:00] [DEBUG] [proxy/service.go:396] ["Proxy init service's parameter table done"] [2023/12/26 07:50:38.300 +00:00] [DEBUG] [proxy/service.go:398] ["Proxy init http server's parameter table done"] [2023/12/26 07:50:38.300 +00:00] [DEBUG] [proxy/service.go:405] ["init Proxy's parameter table done"] ["internal address"=10.244.184.9:19529] ["external address"=10.244.184.9:19530] [2023/12/26 07:50:38.300 +00:00] [DEBUG] [proxy/service.go:408] ["init Proxy's tracer done"] ["service name"="Proxy ip: 10.244.184.9, port: 19530"] [2023/12/26 07:50:38.300 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[cess-etcd-service:2379]"] [minVersion=1.3] [2023/12/26 07:50:38.301 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[cess-etcd-service:2379]"] [minVersion=1.3] [2023/12/26 07:50:38.301 +00:00] [DEBUG] [proxy/service.go:312] ["Proxy internal server listen on tcp"] [port=19529] [2023/12/26 07:50:38.301 +00:00] [DEBUG] [proxy/service.go:319] ["Proxy internal server already listen on tcp"] [port=19529] [2023/12/26 07:50:38.301 +00:00] [DEBUG] [querycoord/service.go:215] [network] [port=19531] [2023/12/26 07:50:38.301 +00:00] [DEBUG] [querynode/service.go:100] [QueryNode] [port=21123] [2023/12/26 07:50:38.301 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[cess-etcd-service:2379]"] [minVersion=1.3] [2023/12/26 07:50:38.301 +00:00] [INFO] [proxy/service.go:435] ["Proxy server listen on tcp"] [port=19530] [2023/12/26 07:50:38.301 +00:00] [INFO] [proxy/service.go:439] ["Proxy server already listen on tcp"] [port=19530] [2023/12/26 07:50:38.301 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[cess-etcd-service:2379]"] [minVersion=1.3] [2023/12/26 07:50:38.301 +00:00] [DEBUG] [proxy/service.go:351] ["create Proxy internal grpc server"] ["enforcement policy"="{"MinTime":5000000000,"PermitWithoutStream":true}"] ["server parameters"="{"MaxConnectionIdle":0,"MaxConnectionAge":0,"MaxConnectionAgeGrace":0,"Time":60000000000,"Timeout":10000000000}"] [2023/12/26 07:50:38.302 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[cess-etcd-service:2379]"] [minVersion=1.3] [2023/12/26 07:50:38.302 +00:00] [DEBUG] [proxy/service.go:223] ["Get proxy rate limiter done"] [port=19530] [2023/12/26 07:50:38.302 +00:00] [DEBUG] [querynode/service.go:117] ["QueryNode connect to etcd successfully"] [2023/12/26 07:50:38.302 +00:00] [INFO] [components/index_coord.go:38] ["IndexCoord running ..."] [2023/12/26 07:50:38.303 +00:00] [INFO] [proxy/service.go:517] ["register Proxy http server"] [2023/12/26 07:50:38.303 +00:00] [DEBUG] [indexnode/indexnode.go:115] ["New IndexNode ..."] [2023/12/26 07:50:38.303 +00:00] [DEBUG] [proxy/service.go:524] ["create RootCoord client for Proxy"] [2023/12/26 07:50:38.303 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.303 +00:00] [DEBUG] [indexnode/service.go:87] [IndexNode] ["network address"=10.244.184.9:21121] ["network port: "=21121] [2023/12/26 07:50:38.303 +00:00] [DEBUG] [datacoord/service.go:147] ["network port"] [port=13333] [2023/12/26 07:50:38.302 +00:00] [DEBUG] [proxy/service.go:287] ["create Proxy grpc server"] ["enforcement policy"="{"MinTime":5000000000,"PermitWithoutStream":true}"] ["server parameters"="{"MaxConnectionIdle":0,"MaxConnectionAge":0,"MaxConnectionAgeGrace":0,"Time":60000000000,"Timeout":10000000000}"] [2023/12/26 07:50:38.304 +00:00] [INFO] [datanode/service.go:246] ["DataNode address"] [address=10.244.184.9:21124] [2023/12/26 07:50:38.307 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.307 +00:00] [DEBUG] [proxy/service.go:530] ["create RootCoord client for Proxy done"] [2023/12/26 07:50:38.307 +00:00] [DEBUG] [proxy/service.go:533] ["Proxy wait for RootCoord to be healthy"] [2023/12/26 07:50:38.308 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:38.308 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:38.310 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=0: node not match"] [2023/12/26 07:50:38.310 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=0: node not match"] [2023/12/26 07:50:38.311 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:38.311 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:38.312 +00:00] [ERROR] [grpcclient/client.go:411] ["retry func failed"] ["retry time"=0] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=0: node not match"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:121\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:37\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:61\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:82\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:534\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:366\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:38.400 +00:00] [DEBUG] [rootcoord/service.go:200] ["grpc init done ..."] [2023/12/26 07:50:38.400 +00:00] [DEBUG] [rootcoord/service.go:203] ["RootCoord start to create DataCoord client"] [2023/12/26 07:50:38.400 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.401 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.401 +00:00] [DEBUG] [rootcoord/service.go:212] ["RootCoord start to create QueryCoord client"] [2023/12/26 07:50:38.401 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.402 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.402 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.402 +00:00] [INFO] [dependency/factory.go:84] ["try to init mq"] [standalone=true] [mqType=rocksmq] [2023/12/26 07:50:38.402 +00:00] [DEBUG] [server/global_rmq.go:39] ["initializing global rmq"] [path=/var/lib/milvus/rdb_data] [2023/12/26 07:50:38.402 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.402 +00:00] [DEBUG] [querycoord/service.go:165] ["QueryCoord try to wait for RootCoord ready"] [2023/12/26 07:50:38.403 +00:00] [DEBUG] [querynode/service.go:127] [QueryNode] [State=Initializing] [2023/12/26 07:50:38.403 +00:00] [INFO] [querynodev2/server.go:269] ["QueryNode session info"] [metaPath=by-dev-milvus/meta] [2023/12/26 07:50:38.403 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.403 +00:00] [INFO] [dependency/factory.go:84] ["try to init mq"] [standalone=true] [mqType=rocksmq] [2023/12/26 07:50:38.403 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[cess-etcd-service:2379]"] [minVersion=1.3] [2023/12/26 07:50:38.403 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:38.403 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:38.404 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.404 +00:00] [INFO] [indexnode/indexnode.go:205] ["IndexNode init"] [state=Initializing] [2023/12/26 07:50:38.404 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.404 +00:00] [INFO] [datanode/service.go:255] ["initializing RootCoord client for DataNode"] [2023/12/26 07:50:38.404 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.404 +00:00] [DEBUG] [sessionutil/session_util.go:292] [getServerID] [reuse=true] [2023/12/26 07:50:38.405 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=0: node not match"] [2023/12/26 07:50:38.405 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=0: node not match"] [2023/12/26 07:50:38.405 +00:00] [DEBUG] [server/rocksmq_impl.go:178] ["Start rocksmq "] ["max proc"=8] [parallism=1] ["lru cache"=1546188226] [2023/12/26 07:50:38.405 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:38.405 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:38.405 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.406 +00:00] [DEBUG] [sessionutil/session_util.go:350] ["Session get serverID success"] [key=id] [ServerId=5] [2023/12/26 07:50:38.406 +00:00] [INFO] [sessionutil/session_util.go:266] ["start server"] [name=querynode] [address=10.244.184.9:21123] [id=5] [2023/12/26 07:50:38.406 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.406 +00:00] [ERROR] [grpcclient/client.go:411] ["retry func failed"] ["retry time"=0] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=0: node not match"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:121\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:37\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:61\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:82\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:166\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:100\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:38.406 +00:00] [INFO] [sessionutil/session_util.go:1167] ["save server info into file"] [content="querynode-5\n"] [filePath=/tmp/milvus/server_id_1] [2023/12/26 07:50:38.406 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:38.406 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:38.406 +00:00] [INFO] [querynodev2/server.go:162] ["QueryNode init session"] [nodeID=5] ["node address"=10.244.184.9:21123] [2023/12/26 07:50:38.406 +00:00] [INFO] [dependency/factory.go:84] ["try to init mq"] [standalone=true] [mqType=rocksmq] [2023/12/26 07:50:38.406 +00:00] [DEBUG] [sessionutil/session_util.go:292] [getServerID] [reuse=true] [2023/12/26 07:50:38.406 +00:00] [INFO] [sessionutil/session_util.go:266] ["start server"] [name=indexnode] [address=10.244.184.9:21121] [id=5] [2023/12/26 07:50:38.407 +00:00] [INFO] [sessionutil/session_util.go:1167] ["save server info into file"] [content="indexnode-5\n"] [filePath=/tmp/milvus/server_id_1] [2023/12/26 07:50:38.407 +00:00] [INFO] [indexnode/indexnode.go:212] ["IndexNode init session successful"] [serverID=5] WARNING: Logging before InitGoogleLogging() is written to STDERR I20231226 07:50:38.407111 40 knowhere_config.cc:94] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384 I20231226 07:50:38.407299 40 knowhere_config.cc:105] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0 I20231226 07:50:38.407356 40 knowhere_config.cc:53] [KNOWHERE][ShowVersion][milvus] Knowhere Version: f4c1757 [2023/12/26 07:50:38.407 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:38.407 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:38.407 +00:00] [INFO] [indexnode/indexnode.go:217] ["Init IndexNode finished"] [] [2023/12/26 07:50:38.407 +00:00] [DEBUG] [indexnode/service.go:74] ["IndexNode init done ..."] [2023/12/26 07:50:38.407 +00:00] [INFO] [indexnode/indexnode.go:229] [IndexNode] [State=Healthy] [2023/12/26 07:50:38.407 +00:00] [INFO] [indexnode/indexnode.go:232] ["IndexNode start finished"] [] [2023/12/26 07:50:38.407 +00:00] [DEBUG] [sessionutil/session_util.go:413] ["service begin to register to etcd"] [serverName=indexnode] [ServerID=5] [2023/12/26 07:50:38.407 +00:00] [DEBUG] [indexnode/task_scheduler.go:242] ["IndexNode TaskScheduler start build loop ..."] [2023/12/26 07:50:38.408 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:38.408 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:38.409 +00:00] [ERROR] [grpcclient/client.go:411] ["retry func failed"] ["retry time"=0] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:121\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:37\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:61\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:82\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:262\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:188\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:38.409 +00:00] [INFO] [sessionutil/session_util.go:442] ["put session key into etcd"] [key=by-dev-milvus/meta/session/indexnode-5] [value="{"ServerID":5,"ServerName":"indexnode","Address":"10.244.184.9:21121","TriggerKill":true,"Version":"2.3.2","IndexEngineVersion":{},"LeaseID":6197389029080642752,"HostName":"milvus-standalone-68bf4b6cd9-9f77n","EnableDisk":true}"] [2023/12/26 07:50:38.409 +00:00] [INFO] [sessionutil/session_util.go:452] ["Service registered successfully"] [ServerName=indexnode] [serverID=5] [2023/12/26 07:50:38.411 +00:00] [DEBUG] [indexnode/service.go:206] ["IndexNode Register etcd success"] [2023/12/26 07:50:38.411 +00:00] [DEBUG] [indexnode/service.go:78] ["IndexNode start done ..."] [2023/12/26 07:50:38.411 +00:00] [DEBUG] [components/index_node.go:55] ["IndexNode successfully started"] [2023/12/26 07:50:38.434 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2023/12/26 07:49:43.511 +00:00] [save=2023/12/26 07:50:41.434 +00:00] [next=2023/12/26 07:50:38.434 +00:00] [2023/12/26 07:50:38.435 +00:00] [INFO] [msgstream/mq_factory.go:18] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data] [2023/12/26 07:50:38.435 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.435 +00:00] [DEBUG] [server/rocksmq_retention.go:80] ["Rocksmq retention goroutine start!"] [2023/12/26 07:50:38.435 +00:00] [INFO] [msgstream/mq_factory.go:18] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data] [2023/12/26 07:50:38.435 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.435 +00:00] [INFO] [msgstream/mq_factory.go:18] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data] [2023/12/26 07:50:38.435 +00:00] [INFO] [querynodev2/server.go:323] ["queryNode try to connect etcd success"] [MetaRootPath=by-dev-milvus/meta] [2023/12/26 07:50:38.435 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:27] ["query node use concurrent safe scheduler"] [max_concurrency=8] [2023/12/26 07:50:38.435 +00:00] [INFO] [querynodev2/server.go:329] ["queryNode init scheduler"] [policy=fifo] [2023/12/26 07:50:38.435 +00:00] [INFO] [segments/segment_loader.go:121] ["SegmentLoader created"] [ioPoolSize=64] [2023/12/26 07:50:38.435 +00:00] [INFO] [querynodev2/server.go:247] ["InitChunkCache done"] [dir=/var/lib/milvus/data/chunk_cache] [policy=willneed] [2023/12/26 07:50:38.435 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.436 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.436 +00:00] [DEBUG] [sessionutil/session_util.go:292] [getServerID] [reuse=true] [2023/12/26 07:50:38.436 +00:00] [INFO] [sessionutil/session_util.go:266] ["start server"] [name=rootcoord] [address=10.244.184.9:53100] [id=5] [2023/12/26 07:50:38.436 +00:00] [INFO] [rootcoord/root_coord.go:155] ["update rootcoord state"] [state=Initializing] [2023/12/26 07:50:38.436 +00:00] [DEBUG] [sessionutil/session_util.go:292] [getServerID] [reuse=true] [2023/12/26 07:50:38.436 +00:00] [INFO] [sessionutil/session_util.go:266] ["start server"] [name=indexcoord] [address=10.244.184.9:13333] [id=5] [2023/12/26 07:50:38.436 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.437 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.437 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2023/12/26 07:49:43.514 +00:00] [save=2023/12/26 07:50:41.437 +00:00] [next=2023/12/26 07:50:38.437 +00:00] [2023/12/26 07:50:38.437 +00:00] [INFO] [rootcoord/root_coord.go:398] ["id allocator initialized"] [root_path=by-dev-milvus/kv] [sub_path=gid] [key=idTimestamp] [2023/12/26 07:50:38.437 +00:00] [DEBUG] [sessionutil/session_util.go:292] [getServerID] [reuse=true] [2023/12/26 07:50:38.437 +00:00] [INFO] [sessionutil/session_util.go:266] ["start server"] [name=datacoord] [address=10.244.184.9:13333] [id=5] [2023/12/26 07:50:38.437 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"] [2023/12/26 07:50:38.438 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"] [2023/12/26 07:50:38.438 +00:00] [INFO] [gc/gc_tuner.go:137] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=23192823398] [2023/12/26 07:50:38.438 +00:00] [INFO] [querynodev2/server.go:383] ["query node init successfully"] [queryNodeID=5] [Address=10.244.184.9:21123] [2023/12/26 07:50:38.438 +00:00] [INFO] [datacoord/server.go:552] ["data coordinator connecting to metadata store"] [metaType=etcd] [2023/12/26 07:50:38.438 +00:00] [DEBUG] [querynode/service.go:226] ["QueryNode init done ..."] [2023/12/26 07:50:38.438 +00:00] [INFO] [datacoord/server.go:560] ["data coordinator successfully connected to metadata store"] [metaType=etcd] [2023/12/26 07:50:38.438 +00:00] [INFO] [querynodev2/server.go:404] ["query node start successfully"] [queryNodeID=5] [Address=10.244.184.9:21123] [mmapEnabled=false] [2023/12/26 07:50:38.438 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:211] ["start execute loop"] [2023/12/26 07:50:38.438 +00:00] [DEBUG] [sessionutil/session_util.go:413] ["service begin to register to etcd"] [serverName=querynode] [ServerID=5] [2023/12/26 07:50:38.439 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2023/12/26 07:50:13.537 +00:00] [save=2023/12/26 07:50:41.438 +00:00] [next=2023/12/26 07:50:38.438 +00:00] [2023/12/26 07:50:38.439 +00:00] [INFO] [rootcoord/root_coord.go:422] ["tso allocator initialized"] [root_path=by-dev-milvus/kv] [sub_path=gid] [key=idTimestamp] [2023/12/26 07:50:38.439 +00:00] [INFO] [rootcoord/root_coord.go:341] ["Using etcd as meta storage."] [2023/12/26 07:50:38.439 +00:00] [DEBUG] [rootcoord/suffix_snapshot.go:555] ["suffix snapshot GC goroutine start!"] [2023/12/26 07:50:38.440 +00:00] [INFO] [rootcoord/meta_table.go:150] ["recover databases"] ["num of dbs"=1] [2023/12/26 07:50:38.440 +00:00] [INFO] [sessionutil/session_util.go:442] ["put session key into etcd"] [key=by-dev-milvus/meta/session/querynode-5] [value="{"ServerID":5,"ServerName":"querynode","Address":"10.244.184.9:21123","TriggerKill":true,"Version":"2.3.2","IndexEngineVersion":{"CurrentIndexVersion":1},"LeaseID":6197389029080642767,"HostName":"milvus-standalone-68bf4b6cd9-9f77n"}"] [2023/12/26 07:50:38.440 +00:00] [INFO] [sessionutil/session_util.go:452] ["Service registered successfully"] [ServerName=querynode] [serverID=5] [2023/12/26 07:50:38.441 +00:00] [INFO] [datacoord/meta.go:171] ["DataCoord meta reloadFromKV done"] [duration=2.743973ms] [2023/12/26 07:50:38.441 +00:00] [DEBUG] [querynode/service.go:231] ["QueryNode start done ..."] [2023/12/26 07:50:38.441 +00:00] [DEBUG] [components/query_node.go:57] ["QueryNode successfully started"] [2023/12/26 07:50:38.442 +00:00] [INFO] [datacoord/channel_store.go:171] ["channel store reload done"] [duration=696.096µs] [2023/12/26 07:50:38.442 +00:00] [DEBUG] [sessionutil/session_util.go:596] ["SessionUtil GetSessions "] [prefix=datanode] [key=datanode-4] [address=10.244.184.9:21124] [2023/12/26 07:50:38.443 +00:00] [INFO] [datacoord/server.go:485] ["DataCoord success to get DataNode sessions"] [sessions="{"datanode-4":{"ServerID":4,"ServerName":"datanode","Address":"10.244.184.9:21124","TriggerKill":true,"Version":"2.3.2","IndexEngineVersion":{},"LeaseID":6197389029080642528,"HostName":"milvus-standalone-68bf4b6cd9-9f77n"}}"] [2023/12/26 07:50:38.443 +00:00] [INFO] [datacoord/channel_manager.go:348] ["register node with no reassignment"] ["registered node"=4] [2023/12/26 07:50:38.443 +00:00] [INFO] [datacoord/channel_manager.go:171] ["starting etcd states checker"] [2023/12/26 07:50:38.443 +00:00] [INFO] [datacoord/channel_manager.go:176] ["starting background balance checker"] [2023/12/26 07:50:38.443 +00:00] [INFO] [datacoord/channel_manager.go:179] ["cluster start up"] [nodes="[4]"] [oNodes="[]"] ["old onlines"="[]"] ["new onlines"="[4]"] [offLines="[]"] [2023/12/26 07:50:38.443 +00:00] [INFO] [rootcoord/meta_table.go:188] ["recover collections from db"] [collection_num=0] [partition_num=0] [2023/12/26 07:50:38.444 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-4] [address=10.244.184.9:21121] [2023/12/26 07:50:38.444 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-5] [address=10.244.184.9:21121] [2023/12/26 07:50:38.444 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=4] ["node address"=10.244.184.9:21121] [2023/12/26 07:50:38.444 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=4] [2023/12/26 07:50:38.444 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=4] ["IndexNode num"=1] [2023/12/26 07:50:38.444 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=5] ["node address"=10.244.184.9:21121] [2023/12/26 07:50:38.444 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=5] [2023/12/26 07:50:38.444 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=5] ["IndexNode num"=2] [2023/12/26 07:50:38.445 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=querynode] [key=querynode-4] [address=10.244.184.9:21123] [2023/12/26 07:50:38.445 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=querynode] [key=querynode-5] [address=10.244.184.9:21123] [2023/12/26 07:50:38.445 +00:00] [INFO] [datacoord/index_engine_version_manager.go:55] ["addOrUpdate version"] [nodeId=4] [minimal=0] [current=1] [2023/12/26 07:50:38.445 +00:00] [INFO] [datacoord/index_engine_version_manager.go:55] ["addOrUpdate version"] [nodeId=5] [minimal=0] [current=1] [2023/12/26 07:50:38.445 +00:00] [INFO] [rootcoord/meta_table.go:204] ["RootCoord meta table reload done"] [duration=5.957223ms] [2023/12/26 07:50:38.445 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.445 +00:00] [INFO] [datacoord/garbage_collector.go:67] ["GC with option"] [enabled=true] [interval=1h0m0s] [missingTolerance=1h0m0s] [dropTolerance=3h0m0s] [2023/12/26 07:50:38.445 +00:00] [DEBUG] [datacoord/service.go:252] ["DataCoord init done ..."] [2023/12/26 07:50:38.445 +00:00] [DEBUG] [sessionutil/session_util.go:413] ["service begin to register to etcd"] [serverName=indexcoord] [ServerID=5] [2023/12/26 07:50:38.445 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_0] [2023/12/26 07:50:38.445 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.445 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_1] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_2] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_3] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_4] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_5] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_6] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_7] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_8] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_9] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_10] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_11] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_12] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_13] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_14] [2023/12/26 07:50:38.446 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true] [2023/12/26 07:50:38.446 +00:00] [WARN] [server/rocksmq_impl.go:399] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_15] [2023/12/26 07:50:38.446 +00:00] [INFO] [rootcoord/dml_channels.go:215] ["init dml channels"] [prefix=by-dev-rootcoord-dml] [num=16] [2023/12/26 07:50:38.446 +00:00] [DEBUG] [rootcoord/root_coord.go:491] ["RootCoord init QuotaCenter done"] [2023/12/26 07:50:38.446 +00:00] [ERROR] [sessionutil/session_util.go:455] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: indexcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:250\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:205\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:254\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:50\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:38.448 +00:00] [INFO] [rootcoord/meta_table.go:1234] ["role already exists"] [role=admin] [2023/12/26 07:50:38.449 +00:00] [INFO] [rootcoord/meta_table.go:1234] ["role already exists"] [role=public] [2023/12/26 07:50:38.453 +00:00] [DEBUG] [rootcoord/service.go:151] ["RootCoord init done ..."] [2023/12/26 07:50:38.453 +00:00] [INFO] [rootcoord/service.go:291] ["RootCoord Core start ..."] [2023/12/26 07:50:38.453 +00:00] [DEBUG] [sessionutil/session_util.go:413] ["service begin to register to etcd"] [serverName=rootcoord] [ServerID=5] [2023/12/26 07:50:38.454 +00:00] [ERROR] [sessionutil/session_util.go:455] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:271\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:292\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:153\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:38.485 +00:00] [INFO] [sessionutil/session_util.go:859] ["register session success"] [role=querynode] [key=by-dev-milvus/meta/session/querynode-5] [2023/12/26 07:50:38.485 +00:00] [INFO] [sessionutil/session_util.go:859] ["register session success"] [role=indexnode] [key=by-dev-milvus/meta/session/indexnode-5] [2023/12/26 07:50:38.513 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:38.513 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:38.608 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:38.608 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:38.611 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:38.611 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:38.914 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:38.915 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.009 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.009 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.011 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.011 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.716 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.716 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.717 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:39.717 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:39.810 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.810 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.811 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:39.811 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:39.812 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.812 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:39.813 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:39.813 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:41.319 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:41.319 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:41.320 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:41.320 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:41.321 +00:00] [ERROR] [grpcclient/client.go:411] ["retry func failed"] ["retry time"=4] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:121\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:37\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:61\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:82\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:534\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:366\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:41.413 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:41.414 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:41.414 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:41.415 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:41.415 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:41.415 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:41.417 +00:00] [ERROR] [grpcclient/client.go:411] ["retry func failed"] ["retry time"=4] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:121\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:37\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:61\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:82\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:166\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:100\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:41.418 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:41.418 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:41.419 +00:00] [ERROR] [grpcclient/client.go:411] ["retry func failed"] ["retry time"=4] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:121\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:37\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:61\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:82\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:262\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:188\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:41.456 +00:00] [ERROR] [sessionutil/session_util.go:455] ["retry func failed"] ["retry time"=4] [error="function CompareAndSwap error for compare is false for key: indexcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:250\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:205\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:254\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:50\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:41.462 +00:00] [ERROR] [sessionutil/session_util.go:455] ["retry func failed"] ["retry time"=4] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:271\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:292\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:153\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:43.297 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev-milvus/config] [endpoints="[cess-etcd-service:2379]"] [2023/12/26 07:50:44.522 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:44.522 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:44.523 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:44.523 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:44.619 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:44.619 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:44.619 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:44.619 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:44.620 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:44.620 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:44.620 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:44.620 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:48.297 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev-milvus/config] [endpoints="[cess-etcd-service:2379]"] [2023/12/26 07:50:50.926 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:50.926 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:50.927 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:50.927 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:51.022 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:51.022 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:51.022 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:51.022 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = expectedNodeID=4, actualNodeID=5: node not match"] [2023/12/26 07:50:51.023 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:51.023 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:51.023 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.244.184.9:53100] [2023/12/26 07:50:51.023 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.244.184.9:53100] [serverID=4] [2023/12/26 07:50:53.297 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev-milvus/config] [endpoints="[cess-etcd-service:2379]"] [2023/12/26 07:50:53.466 +00:00] [ERROR] [sessionutil/session_util.go:455] ["retry func failed"] ["retry time"=8] [error="function CompareAndSwap error for compare is false for key: indexcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:250\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:205\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:254\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:50\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:53.470 +00:00] [ERROR] [sessionutil/session_util.go:455] ["retry func failed"] ["retry time"=8] [error="function CompareAndSwap error for compare is false for key: rootcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Register\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:271\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:292\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:153\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"] [2023/12/26 07:50:58.298 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=by-dev-milvus/config] [endpoints="[cess-etcd-service:2379]"] [2023/12/26 07:51:00.661 +00:00] [DEBUG] [sessionutil/session_util.go:741] ["watch services"] ["delete kv"=] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x3421721]

goroutine 583 [running]: panic({0x41ab5e0, 0x65d94b0}) /usr/local/go/src/runtime/panic.go:987 +0x3bb fp=0xc000dcbaf8 sp=0xc000dcba38 pc=0x1783c1b runtime.panicmem(...) /usr/local/go/src/runtime/panic.go:260 runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:841 +0x37d fp=0xc000dcbb58 sp=0xc000dcbaf8 pc=0x179be5d github.com/milvus-io/milvus/internal/util/sessionutil.(*sessionWatcher).handleWatchResponse(0xc00046de00, {{0x77204a45869fae2f, 0xda15d30cf00f5601, 0x137c, 0x2, {}, {0x0, 0x0, 0x0}, 0x0}, ...}) /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:743 +0x381 fp=0xc000dcbe88 sp=0xc000dcbb58 pc=0x3421721 github.com/milvus-io/milvus/internal/util/sessionutil.(*sessionWatcher).start.func1() /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:665 +0x85 fp=0xc000dcbfe0 sp=0xc000dcbe88 pc=0x3420b25 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000dcbfe8 sp=0xc000dcbfe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/util/sessionutil.(*sessionWatcher).start /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:655 +0x56

goroutine 1 [semacquire]: runtime.gopark(0x0?, 0xc00115b7e8?, 0x20?, 0x2d?, 0xc0006760e0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00115b7a8 sp=0xc00115b788 pc=0x17870d6 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:387 runtime.semacquire1(0xc0003a3a78, 0x70?, 0x1, 0x0, 0x1?) /usr/local/go/src/runtime/sema.go:160 +0x20f fp=0xc00115b810 sp=0xc00115b7a8 pc=0x1798eaf sync.runtime_Semacquire(0x15?) /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc00115b848 sp=0xc00115b810 pc=0x17b8f27 sync.(*WaitGroup).Wait(0xc000cff240?) /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc00115b870 sp=0xc00115b848 pc=0x17ded4b github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc000e4e100) /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:381 +0x946 fp=0xc00115bc70 sp=0xc00115b870 pc=0x3d40686 github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0x10?, {0xc000052180?, 0x3, 0x3}, 0xc000e754a0) /go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:44 +0x2ab fp=0xc00115bd40 sp=0xc00115bc70 pc=0x3d4b8cb github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc000052180?, 0x3, 0x3}) /go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x20e fp=0xc00115bdb8 sp=0xc00115bd40 pc=0x3d4b58e main.main() /go/src/github.com/milvus-io/milvus/cmd/main.go:95 +0x3c5 fp=0xc00115bf80 sp=0xc00115bdb8 pc=0x3d500c5 runtime.main() /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc00115bfe0 sp=0xc00115bf80 pc=0x1786ca7 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00115bfe8 sp=0xc00115bfe0 pc=0x17bd9a1

goroutine 2 [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000c0fb0 sp=0xc0000c0f90 pc=0x17870d6 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:387 runtime.forcegchelper() /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc0000c0fe0 sp=0xc0000c0fb0 pc=0x1786f10 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000c0fe8 sp=0xc0000c0fe0 pc=0x17bd9a1 created by runtime.init.6 /usr/local/go/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000c1780 sp=0xc0000c1760 pc=0x17870d6 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:387 runtime.bgsweep(0x0?) /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000c17c8 sp=0xc0000c1780 pc=0x176f85e runtime.gcenable.func1() /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000c17e0 sp=0xc0000c17c8 pc=0x1764926 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000c17e8 sp=0xc0000c17e0 pc=0x17bd9a1 created by runtime.gcenable /usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]: runtime.gopark(0x35241231fad34?, 0x621da8?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000c1f70 sp=0xc0000c1f50 pc=0x17870d6 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:387 runtime.(*scavengerState).park(0x670ea80) /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc0000c1fa0 sp=0xc0000c1f70 pc=0x176d693 runtime.bgscavenge(0x0?) /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc0000c1fc8 sp=0xc0000c1fa0 pc=0x176dca5 runtime.gcenable.func2() /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc0000c1fe0 sp=0xc0000c1fc8 pc=0x17648c6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000c1fe8 sp=0xc0000c1fe0 pc=0x17bd9a1 created by runtime.gcenable /usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]: runtime.gopark(0x1a0?, 0x6710500?, 0x60?, 0x78?, 0xc0000c0770?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000c0628 sp=0xc0000c0608 pc=0x17870d6 runtime.runfinq() /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000c07e0 sp=0xc0000c0628 pc=0x1763967 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000c07e8 sp=0xc0000c07e0 pc=0x17bd9a1 created by runtime.createfing /usr/local/go/src/runtime/mfinal.go:163 +0x45

goroutine 6 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000c2750 sp=0xc0000c2730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000c27e0 sp=0xc0000c2750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000c27e8 sp=0xc0000c27e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 7 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000c2f50 sp=0xc0000c2f30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000c2fe0 sp=0xc0000c2f50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000c2fe8 sp=0xc0000c2fe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 18 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bc750 sp=0xc0000bc730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000bc7e0 sp=0xc0000bc750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bc7e8 sp=0xc0000bc7e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 8 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000c3750 sp=0xc0000c3730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000c37e0 sp=0xc0000c3750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000c37e8 sp=0xc0000c37e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 34 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000114750 sp=0xc000114730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001147e0 sp=0xc000114750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001147e8 sp=0xc0001147e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 35 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000114f50 sp=0xc000114f30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000114fe0 sp=0xc000114f50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000114fe8 sp=0xc000114fe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 19 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bcf50 sp=0xc0000bcf30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000bcfe0 sp=0xc0000bcf50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bcfe8 sp=0xc0000bcfe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 20 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bd750 sp=0xc0000bd730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000bd7e0 sp=0xc0000bd750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bd7e8 sp=0xc0000bd7e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 36 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000115750 sp=0xc000115730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001157e0 sp=0xc000115750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001157e8 sp=0xc0001157e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 9 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000c3f50 sp=0xc0000c3f30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000c3fe0 sp=0xc0000c3f50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000c3fe8 sp=0xc0000c3fe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 37 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000115f50 sp=0xc000115f30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000115fe0 sp=0xc000115f50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000115fe8 sp=0xc000115fe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 10 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000110750 sp=0xc000110730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001107e0 sp=0xc000110750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001107e8 sp=0xc0001107e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 38 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000116750 sp=0xc000116730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001167e0 sp=0xc000116750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001167e8 sp=0xc0001167e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 11 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000110f50 sp=0xc000110f30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000110fe0 sp=0xc000110f50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000110fe8 sp=0xc000110fe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 39 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000116f50 sp=0xc000116f30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000116fe0 sp=0xc000116f50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000116fe8 sp=0xc000116fe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 21 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bdf50 sp=0xc0000bdf30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000bdfe0 sp=0xc0000bdf50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bdfe8 sp=0xc0000bdfe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 40 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000117750 sp=0xc000117730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001177e0 sp=0xc000117750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001177e8 sp=0xc0001177e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 22 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000be750 sp=0xc0000be730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000be7e0 sp=0xc0000be750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000be7e8 sp=0xc0000be7e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 41 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000117f50 sp=0xc000117f30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000117fe0 sp=0xc000117f50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 12 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000111750 sp=0xc000111730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001117e0 sp=0xc000111750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001117e8 sp=0xc0001117e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 23 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bef50 sp=0xc0000bef30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000befe0 sp=0xc0000bef50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000befe8 sp=0xc0000befe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 13 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000111f50 sp=0xc000111f30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000111fe0 sp=0xc000111f50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 42 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00011e750 sp=0xc00011e730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00011e7e0 sp=0xc00011e750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00011e7e8 sp=0xc00011e7e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 14 [GC worker (idle)]: runtime.gopark(0x3524124bb1039?, 0x1?, 0x32?, 0x50?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000112750 sp=0xc000112730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001127e0 sp=0xc000112750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001127e8 sp=0xc0001127e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 15 [GC worker (idle)]: runtime.gopark(0x35241264e4229?, 0x1?, 0x8c?, 0xa7?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000112f50 sp=0xc000112f30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000112fe0 sp=0xc000112f50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000112fe8 sp=0xc000112fe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 16 [GC worker (idle)]: runtime.gopark(0x675a380?, 0x1?, 0x6e?, 0x6a?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000113750 sp=0xc000113730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001137e0 sp=0xc000113750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001137e8 sp=0xc0001137e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 24 [GC worker (idle)]: runtime.gopark(0x352412654b2e2?, 0x1?, 0x91?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bf750 sp=0xc0000bf730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000bf7e0 sp=0xc0000bf750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bf7e8 sp=0xc0000bf7e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 25 [GC worker (idle)]: runtime.gopark(0x47fe0a0?, 0xc00010e120?, 0x1a?, 0x14?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bff50 sp=0xc0000bff30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000bffe0 sp=0xc0000bff50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bffe8 sp=0xc0000bffe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 26 [GC worker (idle)]: runtime.gopark(0x352412656a7e9?, 0x1?, 0x9a?, 0x22?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00011a750 sp=0xc00011a730 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00011a7e0 sp=0xc00011a750 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00011a7e8 sp=0xc00011a7e0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 43 [GC worker (idle)]: runtime.gopark(0x352412655fd98?, 0x1?, 0x99?, 0x58?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00011ef50 sp=0xc00011ef30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00011efe0 sp=0xc00011ef50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00011efe8 sp=0xc00011efe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 50 [GC worker (idle)]: runtime.gopark(0x3524124b8128d?, 0x1?, 0xfb?, 0xd4?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000113f50 sp=0xc000113f30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000113fe0 sp=0xc000113f50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000113fe8 sp=0xc000113fe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 27 [GC worker (idle)]: runtime.gopark(0x352412654195a?, 0x3?, 0x1?, 0xf8?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00011af50 sp=0xc00011af30 pc=0x17870d6 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00011afe0 sp=0xc00011af50 pc=0x1766691 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00011afe8 sp=0xc00011afe0 pc=0x17bd9a1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 157 [select, locked to thread]: runtime.gopark(0xc0001217a8?, 0x2?, 0x52?, 0x74?, 0xc0001217a4?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000121618 sp=0xc0001215f8 pc=0x17870d6 runtime.selectgo(0xc0001217a8, 0xc0001217a0, 0x0?, 0x0, 0xc000121701?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000121758 sp=0xc000121618 pc=0x1797bfe runtime.ensureSigM.func1() /usr/local/go/src/runtime/signal_unix.go:1004 +0x1b0 fp=0xc0001217e0 sp=0xc000121758 pc=0x17b2dd0 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001217e8 sp=0xc0001217e0 pc=0x17bd9a1 created by runtime.ensureSigM /usr/local/go/src/runtime/signal_unix.go:987 +0xbd

goroutine 144 [select]: runtime.gopark(0xc000120f80?, 0x2?, 0x28?, 0x6b?, 0xc000120f54?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000120dc0 sp=0xc000120da0 pc=0x17870d6 runtime.selectgo(0xc000120f80, 0xc000120f50, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000120f00 sp=0xc000120dc0 pc=0x1797bfe github.com/panjf2000/ants/v2.(*Pool).purgeStaleWorkers(0xc00004e6e0, {0x4bfad98, 0xc0000f8c80}) /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:83 +0x10a fp=0xc000120fb8 sp=0xc000120f00 pc=0x2c0246a github.com/panjf2000/ants/v2.(*Pool).goPurge.func1() /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:147 +0x2e fp=0xc000120fe0 sp=0xc000120fb8 pc=0x2c02a6e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000120fe8 sp=0xc000120fe0 pc=0x17bd9a1 created by github.com/panjf2000/ants/v2.(*Pool).goPurge /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:147 +0xe5

goroutine 145 [select]: runtime.gopark(0xc000121f80?, 0x2?, 0x4b?, 0x3?, 0xc000121f44?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000121dc8 sp=0xc000121da8 pc=0x17870d6 runtime.selectgo(0xc000121f80, 0xc000121f40, 0xc001ea2b70?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000121f08 sp=0xc000121dc8 pc=0x1797bfe github.com/panjf2000/ants/v2.(*Pool).ticktock(0xc00004e6e0, {0x4bfad98, 0xc0000f8cd0}) /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:125 +0x145 fp=0xc000121fb8 sp=0xc000121f08 pc=0x2c02805 github.com/panjf2000/ants/v2.(*Pool).goTicktock.func1() /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:154 +0x2e fp=0xc000121fe0 sp=0xc000121fb8 pc=0x2c02c0e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000121fe8 sp=0xc000121fe0 pc=0x17bd9a1 created by github.com/panjf2000/ants/v2.(*Pool).goTicktock /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:154 +0x115

goroutine 186 [syscall]: runtime.notetsleepg(0x178e3c5?, 0xc00011b7d0?) /usr/local/go/src/runtime/lock_futex.go:236 +0x34 fp=0xc00011b7a0 sp=0xc00011b768 pc=0x17554b4 os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc00011b7c0 sp=0xc00011b7a0 pc=0x17b974f os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc00011b7e0 sp=0xc00011b7c0 pc=0x1887db9 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00011b7e8 sp=0xc00011b7e0 pc=0x17bd9a1 created by os/signal.Notify.func1.1 /usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 158 [select]: runtime.gopark(0xc00011f768?, 0x2?, 0x0?, 0x0?, 0xc00011f714?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00011f598 sp=0xc00011f578 pc=0x17870d6 runtime.selectgo(0xc00011f768, 0xc00011f710, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00011f6d8 sp=0xc00011f598 pc=0x1797bfe github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).handleSignals.func1() /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:278 +0xf0 fp=0xc00011f7e0 sp=0xc00011f6d8 pc=0x3d3fa90 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00011f7e8 sp=0xc00011f7e0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).handleSignals /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:275 +0x165

goroutine 159 [select]: runtime.gopark(0xc00192fe78?, 0x2?, 0xa4?, 0x1?, 0xc00192fe34?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00192fcb8 sp=0xc00192fc98 pc=0x17870d6 runtime.selectgo(0xc00192fe78, 0xc00192fe30, 0xc000336000?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00192fdf8 sp=0xc00192fcb8 pc=0x1797bfe github.com/milvus-io/milvus/pkg/config.(*refresher).refreshPeriodically(0xc00064cae0, {0x46baee9, 0xa}) /go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:68 +0x1fd fp=0xc00192ffb8 sp=0xc00192fdf8 pc=0x2499efd github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1.1() /go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:50 +0x2e fp=0xc00192ffe0 sp=0xc00192ffb8 pc=0x2499bee runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00192ffe8 sp=0xc00192ffe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1 /go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:50 +0xae

goroutine 160 [select]: runtime.gopark(0xc000dabf98?, 0x2?, 0x2?, 0x0?, 0xc000dabf7c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000d75e00 sp=0xc000d75de0 pc=0x17870d6 runtime.selectgo(0xc000d75f98, 0xc000dabf78, 0xc00058eb58?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000d75f40 sp=0xc000d75e00 pc=0x1797bfe google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000487580) /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x73 fp=0xc000d75fc8 sp=0xc000d75f40 pc=0x1e8c193 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x26 fp=0xc000d75fe0 sp=0xc000d75fc8 pc=0x1e8c0e6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000d75fe8 sp=0xc000d75fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x22a

goroutine 212 [select]: runtime.gopark(0xc000db4780?, 0x2?, 0xca?, 0x69?, 0xc000db4744?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000db45c8 sp=0xc000db45a8 pc=0x17870d6 runtime.selectgo(0xc000db4780, 0xc000db4740, 0xc0010f51e8?, 0x0, 0xc000f3eaa0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000db4708 sp=0xc000db45c8 pc=0x1797bfe github.com/panjf2000/ants/v2.(*Pool).ticktock(0xc0002e7d60, {0x4bfad98, 0xc000d3f770}) /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:125 +0x145 fp=0xc000db47b8 sp=0xc000db4708 pc=0x2c02805 github.com/panjf2000/ants/v2.(*Pool).goTicktock.func1() /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:154 +0x2e fp=0xc000db47e0 sp=0xc000db47b8 pc=0x2c02c0e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000db47e8 sp=0xc000db47e0 pc=0x17bd9a1 created by github.com/panjf2000/ants/v2.(*Pool).goTicktock /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:154 +0x115

goroutine 176 [select]: runtime.gopark(0xc0012dbf08?, 0x2?, 0xa3?, 0x50?, 0xc0012dbedc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012dbd68 sp=0xc0012dbd48 pc=0x17870d6 runtime.selectgo(0xc0012dbf08, 0xc0012dbed8, 0x1a95869?, 0x0, 0xc00160a000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012dbea8 sp=0xc0012dbd68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00030e5a0, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc0012dbf38 sp=0xc0012dbea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00161e000) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc0012dbf98 sp=0xc0012dbf38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x7b fp=0xc0012dbfe0 sp=0xc0012dbf98 pc=0x1e0f4bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012dbfe8 sp=0xc0012dbfe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x23fc

goroutine 175 [IO wait]: runtime.gopark(0xc001d48418?, 0xb?, 0x0?, 0x0?, 0xe?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000466ad8 sp=0xc000466ab8 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000466b10 sp=0xc000466ad8 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8f3b8, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000466b30 sp=0xc000466b10 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc00035a100?, 0xc001602000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000466b58 sp=0xc000466b30 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00035a100, {0xc001602000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000466bf0 sp=0xc000466b58 pc=0x1842cd9 net.(*netFD).Read(0xc00035a100, {0xc001602000?, 0x1060100000000?, 0x8?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000466c38 sp=0xc000466bf0 pc=0x1a69a29 net.(*conn).Read(0xc000334030, {0xc001602000?, 0xc000669410?, 0xc00161a000?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000466c80 sp=0xc000466c38 pc=0x1a7ea05 net.(*TCPConn).Read(0x801010601?, {0xc001602000?, 0xc000466d28?, 0x801010601?}) :1 +0x29 fp=0xc000466cb0 sp=0xc000466c80 pc=0x1a95569 bufio.(*Reader).Read(0xc000e75da0, {0xc00033a200, 0x9, 0x7fcba69aaf18?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000466ce8 sp=0xc000466cb0 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc000e75da0}, {0xc00033a200, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000466d30 sp=0xc000466ce8 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc00033a200?, 0x9?, 0x300ca8035?}, {0x4bd2e40?, 0xc000e75da0?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc000466d80 sp=0xc000466d30 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc00033a1c0) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc000466e30 sp=0xc000466d80 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00161c000, 0x1?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x257 fp=0xc000466fc0 sp=0xc000466e30 pc=0x1e19d37 google.golang.org/grpc/internal/transport.newHTTP2Client.func11() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2a fp=0xc000466fe0 sp=0xc000466fc0 pc=0x1e0f5ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000466fe8 sp=0xc000466fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1e2a

goroutine 227 [IO wait]: runtime.gopark(0xc000431a98?, 0xb?, 0x0?, 0x0?, 0x7?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000d3ad8 sp=0xc0000d3ab8 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0000d3b10 sp=0xc0000d3ad8 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8fe08, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0000d3b30 sp=0xc0000d3b10 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc000552100?, 0xc000de0000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0000d3b58 sp=0xc0000d3b30 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc000552100, {0xc000de0000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0000d3bf0 sp=0xc0000d3b58 pc=0x1842cd9 net.(*netFD).Read(0xc000552100, {0xc000de0000?, 0x1060100000000?, 0x8?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0000d3c38 sp=0xc0000d3bf0 pc=0x1a69a29 net.(*conn).Read(0xc000556038, {0xc000de0000?, 0xc000447a80?, 0xc0004c4140?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0000d3c80 sp=0xc0000d3c38 pc=0x1a7ea05 net.(*TCPConn).Read(0x801010601?, {0xc000de0000?, 0xc0000d3d28?, 0x801010601?}) :1 +0x29 fp=0xc0000d3cb0 sp=0xc0000d3c80 pc=0x1a95569 bufio.(*Reader).Read(0xc0003029c0, {0xc000dae040, 0x9, 0x7fcba69aa5b8?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0000d3ce8 sp=0xc0000d3cb0 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc0003029c0}, {0xc000dae040, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0000d3d30 sp=0xc0000d3ce8 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc000dae040?, 0x9?, 0x4b2768b3b?}, {0x4bd2e40?, 0xc0003029c0?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc0000d3d80 sp=0xc0000d3d30 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc000dae000) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc0000d3e30 sp=0xc0000d3d80 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000dbc000, 0x0?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x257 fp=0xc0000d3fc0 sp=0xc0000d3e30 pc=0x1e19d37 google.golang.org/grpc/internal/transport.newHTTP2Client.func11() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2a fp=0xc0000d3fe0 sp=0xc0000d3fc0 pc=0x1e0f5ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000d3fe8 sp=0xc0000d3fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1e2a

goroutine 228 [select]: runtime.gopark(0xc001cd5f08?, 0x2?, 0xa3?, 0x50?, 0xc001cd5edc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001cd5d68 sp=0xc001cd5d48 pc=0x17870d6 runtime.selectgo(0xc001cd5f08, 0xc001cd5ed8, 0x1a95869?, 0x0, 0xc000e56000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001cd5ea8 sp=0xc001cd5d68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00054c500, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc001cd5f38 sp=0xc001cd5ea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000dcc000) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc001cd5f98 sp=0xc001cd5f38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x7b fp=0xc001cd5fe0 sp=0xc001cd5f98 pc=0x1e0f4bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001cd5fe8 sp=0xc001cd5fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x23fc

goroutine 229 [select]: runtime.gopark(0xc0014e5e78?, 0x2?, 0x0?, 0x0?, 0xc0014e5e34?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0014e5cb8 sp=0xc0014e5c98 pc=0x17870d6 runtime.selectgo(0xc0014e5e78, 0xc0014e5e30, 0xc000f3a680?, 0x0, 0xc000db96dc?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0014e5df8 sp=0xc0014e5cb8 pc=0x1797bfe github.com/milvus-io/milvus/pkg/config.(*refresher).refreshPeriodically(0xc00064cfc0, {0x46baea3, 0xa}) /go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:68 +0x1fd fp=0xc0014e5fb8 sp=0xc0014e5df8 pc=0x2499efd github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1.1() /go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:50 +0x2e fp=0xc0014e5fe0 sp=0xc0014e5fb8 pc=0x2499bee runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0014e5fe8 sp=0xc0014e5fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1 /go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:50 +0xae

goroutine 230 [select]: runtime.gopark(0xc001463e78?, 0x2?, 0xa4?, 0x1?, 0xc001463e34?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001463cb8 sp=0xc001463c98 pc=0x17870d6 runtime.selectgo(0xc001463e78, 0xc001463e30, 0xc000f3a6c0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001463df8 sp=0xc001463cb8 pc=0x1797bfe github.com/milvus-io/milvus/pkg/config.(*refresher).refreshPeriodically(0xc000302ba0, {0x46baee9, 0xa}) /go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:68 +0x1fd fp=0xc001463fb8 sp=0xc001463df8 pc=0x2499efd github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1.1() /go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:50 +0x2e fp=0xc001463fe0 sp=0xc001463fb8 pc=0x2499bee runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001463fe8 sp=0xc001463fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1 /go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:50 +0xae

goroutine 231 [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000cfb18 sp=0xc0000cfaf8 pc=0x17870d6 runtime.netpollblock(0x7fcba69b6a88?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0000cfb50 sp=0xc0000cfb18 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8fd18, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0000cfb70 sp=0xc0000cfb50 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc000f5a300?, 0xc0000cfc58?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0000cfb98 sp=0xc0000cfb70 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc000f5a300) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0000cfc40 sp=0xc0000cfb98 pc=0x18471fd net.(*netFD).accept(0xc000f5a300) /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc0000cfcf8 sp=0xc0000cfc40 pc=0x1a6bc15 net.(*TCPListener).accept(0xc000ede4b0) /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc0000cfd20 sp=0xc0000cfcf8 pc=0x1a8a025 net.(*TCPListener).Accept(0xc000ede4b0) /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc0000cfd50 sp=0xc0000cfd20 pc=0x1a88fbd net/http.(*onceCloseListener).Accept(0x4bfadd0?) :1 +0x2a fp=0xc0000cfd68 sp=0xc0000cfd50 pc=0x1bc3aca net/http.(*Server).Serve(0xc000f1c1e0, {0x4bf8310, 0xc000ede4b0}) /usr/local/go/src/net/http/server.go:3059 +0x385 fp=0xc0000cfe98 sp=0xc0000cfd68 pc=0x1b98345 net/http.(*Server).ListenAndServe(0xc000f1c1e0) /usr/local/go/src/net/http/server.go:2988 +0x7d fp=0xc0000cfec8 sp=0xc0000cfe98 pc=0x1b97ebd github.com/milvus-io/milvus/internal/http.ServeHTTP.func1() /go/src/github.com/milvus-io/milvus/internal/http/server.go:79 +0x15a fp=0xc0000cffe0 sp=0xc0000cfec8 pc=0x3b6ad9a runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000cffe8 sp=0xc0000cffe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/http.ServeHTTP /go/src/github.com/milvus-io/milvus/internal/http/server.go:75 +0x25

goroutine 235 [select]: runtime.gopark(0xc0012119a0?, 0x2?, 0x0?, 0x33?, 0xc00121194c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012117a8 sp=0xc001211788 pc=0x17870d6 runtime.selectgo(0xc0012119a0, 0xc001211948, 0x4bd3f60?, 0x0, 0xb?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012118e8 sp=0xc0012117a8 pc=0x1797bfe github.com/milvus-io/milvus/pkg/util/retry.Do({0x4bfad98, 0xc00054dc20}, 0xc001211af8, {0xc001211af0, 0x1, 0x0?}) /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:56 +0x4f1 fp=0xc001211a90 sp=0xc0012118e8 pc=0x2655a51 github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService(0xc00129d6c0) /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455 +0x2ae fp=0xc001211bb0 sp=0xc001211a90 pc=0x341d0ae github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register(0xc00129d6c0) /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276 +0x33 fp=0xc001211c88 sp=0xc001211bb0 pc=0x341acf3 github.com/milvus-io/milvus/internal/rootcoord.(*Core).Register(0xc001118000) /go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:271 +0x2c fp=0xc001211ce0 sp=0xc001211c88 pc=0x3cff6cc github.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).start(0xc0001f2f30) /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:292 +0x63 fp=0xc001211e48 sp=0xc001211ce0 pc=0x3d38c43 github.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run(0xc0003a3a90?) /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:153 +0x4f fp=0xc001211e88 sp=0xc001211e48 pc=0x3d3752f github.com/milvus-io/milvus/cmd/components.(*RootCoord).Run(0xc000532840?) /go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:52 +0x2f fp=0xc001211f58 sp=0xc001211e88 pc=0x3d3c40f github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113 +0x15a fp=0xc001211fe0 sp=0xc001211f58 pc=0x3d4157a runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001211fe8 sp=0xc001211fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:100 +0x158

goroutine 253 [select]: runtime.gopark(0xc00163e4c0?, 0x2?, 0xc0?, 0x83?, 0xc00163e46c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00163e2c8 sp=0xc00163e2a8 pc=0x17870d6 runtime.selectgo(0xc00163e4c0, 0xc00163e468, 0x4bd3f60?, 0x0, 0xb?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00163e408 sp=0xc00163e2c8 pc=0x1797bfe github.com/milvus-io/milvus/pkg/util/retry.Do({0x4bfad98, 0xc00141d130}, 0xc00163e798, {0xc00163e688, 0x3, 0x40?}) /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:56 +0x4f1 fp=0xc00163e5b0 sp=0xc00163e408 pc=0x2655a51 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call(0x4c4bec0, {0x4bfad98, 0xc00141d130}, 0xc00044d710) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411 +0x5b3 fp=0xc00163e800 sp=0xc00163e5b0 pc=0x346c553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call(0x4c4bec0, {0x4bfad98, 0xc000f3eaa0}, 0xc000100401) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489 +0x96 fp=0xc00163e990 sp=0xc00163e800 pc=0x346bb76 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall(...) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall({0x4bfad98?, 0xc000f3eaa0?}, 0x42779c0?) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:503 +0x3f fp=0xc00163e9c8 sp=0xc00163e990 pc=0x346f2df github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]({0x4bfad98, 0xc000f3eaa0?}, 0xc000662078, 0xc0006620a8) /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 +0xc2 fp=0xc00163ea00 sp=0xc00163e9c8 pc=0x3471782 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates(0xc0004380a0?, {0x4bfad98?, 0xc000f3eaa0}, 0x1756da7?, {0x20?, 0x4347280?, 0x17dcd01?}) /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:121 +0x85 fp=0xc00163ea48 sp=0xc00163ea00 pc=0x34613e5 github.com/milvus-io/milvus/internal/types.RootCoordClient.GetComponentStates({0x4c4d978?, 0xc000662078?}, {0x4bfad98?, 0xc000f3eaa0?}, 0x456a420?, {0x0?, 0x17569ca?, 0x7fcba69aaf18?}) :1 +0x62 fp=0xc00163ea90 sp=0xc00163ea48 pc=0x3419022 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1() /go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:37 +0xca fp=0xc00163eb58 sp=0xc00163ea90 pc=0x3796f6a github.com/milvus-io/milvus/pkg/util/retry.Do({0x4bfad98, 0xc000f3eaa0}, 0xc00163ed60, {0xc001214d50, 0x2, 0x0?}) /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44 +0x148 fp=0xc00163ed00 sp=0xc00163eb58 pc=0x26556a8 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]({0x4bfad98?, 0xc000f3eaa0?}, {0x4c4d978?, 0xc000662078?}, {0x46b8c68?, 0x18?}, {0xc001214e64, 0x1, 0x1}, 0xc00053e220?, ...) /go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:61 +0x145 fp=0xc00163edc8 sp=0xc00163ed00 pc=0x37973c5 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy... /go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:82 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init(0xc00044e000) /go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:534 +0x26fc fp=0xc00163fd20 sp=0xc00163edc8 pc=0x3b71bfc github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run(0xc00044e000) /go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:366 +0x56 fp=0xc00163fe88 sp=0xc00163fd20 pc=0x3b6ee96 github.com/milvus-io/milvus/cmd/components.(*Proxy).Run(0xc00044a000?) /go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:52 +0x2e fp=0xc00163ff58 sp=0xc00163fe88 pc=0x3d3b8ce github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113 +0x15a fp=0xc00163ffe0 sp=0xc00163ff58 pc=0x3d4157a runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00163ffe8 sp=0xc00163ffe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:100 +0x158

goroutine 236 [select]: runtime.gopark(0xc001291f98?, 0x2?, 0x2?, 0x0?, 0xc001291f7c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001291e00 sp=0xc001291de0 pc=0x17870d6 runtime.selectgo(0xc001291f98, 0xc001291f78, 0xc00054e6d8?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001291f40 sp=0xc001291e00 pc=0x1797bfe google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00110c780) /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x73 fp=0xc001291fc8 sp=0xc001291f40 pc=0x1e8c193 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x26 fp=0xc001291fe0 sp=0xc001291fc8 pc=0x1e8c0e6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001291fe8 sp=0xc001291fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x22a

goroutine 579 [select]: runtime.gopark(0xc000db97b0?, 0x2?, 0x87?, 0x71?, 0xc000db9794?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000db9618 sp=0xc000db95f8 pc=0x17870d6 runtime.selectgo(0xc000db97b0, 0xc000db9790, 0x100000001e05560?, 0x0, 0x4bfae40?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000db9758 sp=0xc000db9618 pc=0x1797bfe google.golang.org/grpc.newClientStreamWithParams.func4() /go/pkg/mod/google.golang.org/[email protected]/stream.go:375 +0x92 fp=0xc000db97e0 sp=0xc000db9758 pc=0x1eaea92 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000db97e8 sp=0xc000db97e0 pc=0x17bd9a1 created by google.golang.org/grpc.newClientStreamWithParams /go/pkg/mod/google.golang.org/[email protected]/stream.go:374 +0xf2a

goroutine 211 [select]: runtime.gopark(0xc000dbb780?, 0x2?, 0xca?, 0x69?, 0xc000dbb754?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000dbb5c0 sp=0xc000dbb5a0 pc=0x17870d6 runtime.selectgo(0xc000dbb780, 0xc000dbb750, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000dbb700 sp=0xc000dbb5c0 pc=0x1797bfe github.com/panjf2000/ants/v2.(*Pool).purgeStaleWorkers(0xc0002e7d60, {0x4bfad98, 0xc000d3f720}) /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:83 +0x10a fp=0xc000dbb7b8 sp=0xc000dbb700 pc=0x2c0246a github.com/panjf2000/ants/v2.(*Pool).goPurge.func1() /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:147 +0x2e fp=0xc000dbb7e0 sp=0xc000dbb7b8 pc=0x2c02a6e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000dbb7e8 sp=0xc000dbb7e0 pc=0x17bd9a1 created by github.com/panjf2000/ants/v2.(*Pool).goPurge /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:147 +0xe5

goroutine 254 [select]: runtime.gopark(0xc0000d0f40?, 0x2?, 0x78?, 0xb8?, 0xc0000d0ed4?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000d0d50 sp=0xc0000d0d30 pc=0x17870d6 runtime.selectgo(0xc0000d0f40, 0xc0000d0ed0, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0000d0e90 sp=0xc0000d0d50 pc=0x1797bfe github.com/milvus-io/milvus/internal/proxy.(*LookAsideBalancer).checkQueryNodeHealthLoop(0xc0004406c0, {0x4bfad98?, 0xc000f3eaa0?}) /go/src/github.com/milvus-io/milvus/internal/proxy/look_aside_balancer.go:207 +0x2aa fp=0xc0000d0fb8 sp=0xc0000d0e90 pc=0x3a91bea github.com/milvus-io/milvus/internal/proxy.(*LookAsideBalancer).Start.func1() /go/src/github.com/milvus-io/milvus/internal/proxy/look_aside_balancer.go:79 +0x2e fp=0xc0000d0fe0 sp=0xc0000d0fb8 pc=0x3a90bae runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000d0fe8 sp=0xc0000d0fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/proxy.(*LookAsideBalancer).Start /go/src/github.com/milvus-io/milvus/internal/proxy/look_aside_balancer.go:79 +0xaa

goroutine 255 [select]: runtime.gopark(0xc0000d4ec8?, 0x2?, 0x2?, 0x0?, 0xc0000d4e94?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000d4d18 sp=0xc0000d4cf8 pc=0x17870d6 runtime.selectgo(0xc0000d4ec8, 0xc0000d4e90, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0000d4e58 sp=0xc0000d4d18 pc=0x1797bfe github.com/milvus-io/milvus/pkg/util/resource.(*manager).backgroundGC(0xc000440720) /go/src/github.com/milvus-io/milvus/pkg/util/resource/resource_manager.go:182 +0x10b fp=0xc0000d4fc8 sp=0xc0000d4e58 pc=0x395258b github.com/milvus-io/milvus/pkg/util/resource.NewManager.func1() /go/src/github.com/milvus-io/milvus/pkg/util/resource/resource_manager.go:173 +0x26 fp=0xc0000d4fe0 sp=0xc0000d4fc8 pc=0x3952446 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000d4fe8 sp=0xc0000d4fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/pkg/util/resource.NewManager /go/src/github.com/milvus-io/milvus/pkg/util/resource/resource_manager.go:173 +0x138

goroutine 239 [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000dc9980 sp=0xc000dc9960 pc=0x17870d6 runtime.netpollblock(0x7fcba683aa18?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000dc99b8 sp=0xc000dc9980 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8fc28, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000dc99d8 sp=0xc000dc99b8 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc0010fd180?, 0x4?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000dc9a00 sp=0xc000dc99d8 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc0010fd180) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000dc9aa8 sp=0xc000dc9a00 pc=0x18471fd net.(*netFD).accept(0xc0010fd180) /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000dc9b60 sp=0xc000dc9aa8 pc=0x1a6bc15 net.(*TCPListener).accept(0xc00054e708) /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000dc9b88 sp=0xc000dc9b60 pc=0x1a8a025 net.(*TCPListener).Accept(0xc00054e708) /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000dc9bb8 sp=0xc000dc9b88 pc=0x1a88fbd google.golang.org/grpc.(*Server).Serve(0xc000cc61e0, {0x4bf8310?, 0xc00054e708}) /go/pkg/mod/google.golang.org/[email protected]/server.go:824 +0x475 fp=0xc000dc9d00 sp=0xc000dc9bb8 pc=0x1ea13d5 github.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).startGrpcLoop(0xc0001f2f30, 0xcf6c) /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:285 +0x9eb fp=0xc000dc9fc0 sp=0xc000dc9d00 pc=0x3d3884b github.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).startGrpc.func1() /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:225 +0x2a fp=0xc000dc9fe0 sp=0xc000dc9fc0 pc=0x3d37e2a runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000dc9fe8 sp=0xc000dc9fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).startGrpc /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:225 +0x7f

goroutine 512 [select]: runtime.gopark(0xc0012b1f98?, 0x2?, 0x38?, 0xef?, 0xc0012b1f7c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012b1e00 sp=0xc0012b1de0 pc=0x17870d6 runtime.selectgo(0xc0012b1f98, 0xc0012b1f78, 0xc000013fe0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012b1f40 sp=0xc0012b1e00 pc=0x1797bfe google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000596c80) /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x73 fp=0xc0012b1fc8 sp=0xc0012b1f40 pc=0x1e8c193 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x26 fp=0xc0012b1fe0 sp=0xc0012b1fc8 pc=0x1e8c0e6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012b1fe8 sp=0xc0012b1fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x22a

goroutine 359 [select]: runtime.gopark(0xc0014619a8?, 0x2?, 0x0?, 0x33?, 0xc001461954?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0014617b0 sp=0xc001461790 pc=0x17870d6 runtime.selectgo(0xc0014619a8, 0xc001461950, 0x4bd3f60?, 0x0, 0xb?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0014618f0 sp=0xc0014617b0 pc=0x1797bfe github.com/milvus-io/milvus/pkg/util/retry.Do({0x4bfad98, 0xc000656be0}, 0xc001461b00, {0xc001461af8, 0x1, 0x0?}) /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:56 +0x4f1 fp=0xc001461a98 sp=0xc0014618f0 pc=0x2655a51 github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService(0xc00129d860) /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455 +0x2ae fp=0xc001461bb8 sp=0xc001461a98 pc=0x341d0ae github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register(0xc00129d860) /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276 +0x33 fp=0xc001461c90 sp=0xc001461bb8 pc=0x341acf3 github.com/milvus-io/milvus/internal/datacoord.(*Server).Register(0xc000678000) /go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:250 +0x2c fp=0xc001461cf0 sp=0xc001461c90 pc=0x353c04c github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start(0xc000654de0) /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:205 +0x45 fp=0xc001461e48 sp=0xc001461cf0 pc=0x356d365 github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run(0xc000650004?) /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:254 +0x4f fp=0xc001461e88 sp=0xc001461e48 pc=0x356da6f github.com/milvus-io/milvus/cmd/components.(*DataCoord).Run(0xc00066a000?) /go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:50 +0x2f fp=0xc001461f58 sp=0xc001461e88 pc=0x3d3ac6f github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113 +0x15a fp=0xc001461fe0 sp=0xc001461f58 pc=0x3d4157a runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001461fe8 sp=0xc001461fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:100 +0x158

goroutine 422 [IO wait]: runtime.gopark(0xc0018f8718?, 0xb?, 0x0?, 0x0?, 0x15?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000464ad8 sp=0xc000464ab8 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000464b10 sp=0xc000464ad8 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8f1d8, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000464b30 sp=0xc000464b10 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc001202780?, 0xc0012dc000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000464b58 sp=0xc000464b30 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc001202780, {0xc0012dc000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000464bf0 sp=0xc000464b58 pc=0x1842cd9 net.(*netFD).Read(0xc001202780, {0xc0012dc000?, 0x1060100000000?, 0x8?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000464c38 sp=0xc000464bf0 pc=0x1a69a29 net.(*conn).Read(0xc00128a0b0, {0xc0012dc000?, 0xc000d42948?, 0xc001498140?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000464c80 sp=0xc000464c38 pc=0x1a7ea05 net.(*TCPConn).Read(0x801010601?, {0xc0012dc000?, 0xc000464d28?, 0x801010601?}) :1 +0x29 fp=0xc000464cb0 sp=0xc000464c80 pc=0x1a95569 bufio.(*Reader).Read(0xc00129a540, {0xc00148a3c0, 0x9, 0x7fcba69aaa68?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000464ce8 sp=0xc000464cb0 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc00129a540}, {0xc00148a3c0, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000464d30 sp=0xc000464ce8 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc00148a3c0?, 0x9?, 0x2fb1a05d5?}, {0x4bd2e40?, 0xc00129a540?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc000464d80 sp=0xc000464d30 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc00148a380) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc000464e30 sp=0xc000464d80 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0014b4000, 0xc00122c240?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x257 fp=0xc000464fc0 sp=0xc000464e30 pc=0x1e19d37 google.golang.org/grpc/internal/transport.newHTTP2Client.func11() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2a fp=0xc000464fe0 sp=0xc000464fc0 pc=0x1e0f5ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000464fe8 sp=0xc000464fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1e2a

goroutine 301 [select]: runtime.gopark(0xc001487130?, 0x2?, 0xc0?, 0x83?, 0xc0014870dc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001486f38 sp=0xc001486f18 pc=0x17870d6 runtime.selectgo(0xc001487130, 0xc0014870d8, 0x4bd3f60?, 0x0, 0xb?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001487078 sp=0xc001486f38 pc=0x1797bfe github.com/milvus-io/milvus/pkg/util/retry.Do({0x4bfad98, 0xc001284910}, 0xc001487408, {0xc0014872f8, 0x3, 0x40?}) /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:56 +0x4f1 fp=0xc001487220 sp=0xc001487078 pc=0x2655a51 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call(0x4c4bec0, {0x4bfad98, 0xc001284910}, 0xc001174420) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411 +0x5b3 fp=0xc001487470 sp=0xc001487220 pc=0x346c553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call(0x4c4bec0, {0x4bfad98, 0xc001122b90}, 0xc0000c4801) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489 +0x96 fp=0xc001487600 sp=0xc001487470 pc=0x346bb76 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall(...) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall({0x4bfad98?, 0xc001122b90?}, 0x42779c0?) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:503 +0x3f fp=0xc001487638 sp=0xc001487600 pc=0x346f2df github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]({0x4bfad98, 0xc001122b90?}, 0xc000ede0c0, 0xc000ede0f0) /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 +0xc2 fp=0xc001487670 sp=0xc001487638 pc=0x3471782 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates(0xc000cb2080?, {0x4bfad98?, 0xc001122b90}, 0x1756da7?, {0x20?, 0x4347280?, 0x1?}) /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:121 +0x85 fp=0xc0014876b8 sp=0xc001487670 pc=0x34613e5 github.com/milvus-io/milvus/internal/types.RootCoordClient.GetComponentStates({0x4c4d978?, 0xc000ede0c0?}, {0x4bfad98?, 0xc001122b90?}, 0x456a420?, {0x0?, 0x17569ca?, 0x7fcba69ac1d8?}) :1 +0x62 fp=0xc001487700 sp=0xc0014876b8 pc=0x3419022 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1() /go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:37 +0xca fp=0xc0014877c8 sp=0xc001487700 pc=0x3796f6a github.com/milvus-io/milvus/pkg/util/retry.Do({0x4bfad98, 0xc001122b90}, 0xc0014879d0, {0xc0014879c0, 0x2, 0x0?}) /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44 +0x148 fp=0xc001487970 sp=0xc0014877c8 pc=0x26556a8 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]({0x4bfad98?, 0xc001122b90?}, {0x4c4d978?, 0xc000ede0c0?}, {0x46b8c68?, 0x18?}, {0xc001487a9c, 0x1, 0x1}, 0xc00053e220?, ...) /go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:61 +0x145 fp=0xc001487a38 sp=0xc001487970 pc=0x37973c5 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy... /go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:82 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init(0xc0001f34d0) /go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:166 +0x825 fp=0xc001487e48 sp=0xc001487a38 pc=0x3c120c5 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run(0xc0003a3d90?) /go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:100 +0x25 fp=0xc001487e88 sp=0xc001487e48 pc=0x3c11805 github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run(0xc000532f60?) /go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53 +0x2f fp=0xc001487f58 sp=0xc001487e88 pc=0x3d3bc8f github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113 +0x15a fp=0xc001487fe0 sp=0xc001487f58 pc=0x3d4157a runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001487fe8 sp=0xc001487fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:100 +0x158

goroutine 256 [select]: runtime.gopark(0xc000da7f98?, 0x2?, 0x2?, 0x0?, 0xc000da7f7c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000da7e00 sp=0xc000da7de0 pc=0x17870d6 runtime.selectgo(0xc000da7f98, 0xc000da7f78, 0xc0001568b8?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000da7f40 sp=0xc000da7e00 pc=0x1797bfe google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00046c580) /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x73 fp=0xc000da7fc8 sp=0xc000da7f40 pc=0x1e8c193 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x26 fp=0xc000da7fe0 sp=0xc000da7fc8 pc=0x1e8c0e6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000da7fe8 sp=0xc000da7fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x22a

goroutine 302 [select]: runtime.gopark(0xc0013e5f98?, 0x2?, 0x2?, 0x0?, 0xc0013e5f7c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0013e5e00 sp=0xc0013e5de0 pc=0x17870d6 runtime.selectgo(0xc0013e5f98, 0xc0013e5f78, 0xc00054eea0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0013e5f40 sp=0xc0013e5e00 pc=0x1797bfe google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00110cb00) /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x73 fp=0xc0013e5fc8 sp=0xc0013e5f40 pc=0x1e8c193 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x26 fp=0xc0013e5fe0 sp=0xc0013e5fc8 pc=0x1e8c0e6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0013e5fe8 sp=0xc0013e5fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x22a

goroutine 377 [sleep]: runtime.gopark(0x352412e0f9ccd?, 0x0?, 0xc0?, 0xbc?, 0x17569ff?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000469c48 sp=0xc000469c28 pc=0x17870d6 time.Sleep(0x8bb2c97000) /usr/local/go/src/runtime/time.go:195 +0x135 fp=0xc000469c88 sp=0xc000469c48 pc=0x17ba1d5 github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.NewRocksMQ.func1() /go/src/github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_impl.go:271 +0x7e fp=0xc000469fe0 sp=0xc000469c88 pc=0x34bcf5e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000469fe8 sp=0xc000469fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.NewRocksMQ /go/src/github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_impl.go:269 +0xa2e

goroutine 386 [select]: runtime.gopark(0xc001403f98?, 0x2?, 0x2?, 0x0?, 0xc001403f7c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001403e00 sp=0xc001403de0 pc=0x17870d6 runtime.selectgo(0xc001403f98, 0xc001403f78, 0xc00140a138?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001403f40 sp=0xc001403e00 pc=0x1797bfe google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001416180) /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x73 fp=0xc001403fc8 sp=0xc001403f40 pc=0x1e8c193 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x26 fp=0xc001403fe0 sp=0xc001403fc8 pc=0x1e8c0e6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001403fe8 sp=0xc001403fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x22a

goroutine 486 [select]: runtime.gopark(0xc00066ef90?, 0x2?, 0xca?, 0x69?, 0xc00066ef4c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00066edb8 sp=0xc00066ed98 pc=0x17870d6 runtime.selectgo(0xc00066ef90, 0xc00066ef48, 0x0?, 0x0, 0xc0012782a0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00066eef8 sp=0xc00066edb8 pc=0x1797bfe github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).indexBuildLoop(0xc00067e180) /go/src/github.com/milvus-io/milvus/internal/indexnode/task_scheduler.go:245 +0x125 fp=0xc00066efc8 sp=0xc00066eef8 pc=0x37b29e5 github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).Start.func1() /go/src/github.com/milvus-io/milvus/internal/indexnode/task_scheduler.go:266 +0x26 fp=0xc00066efe0 sp=0xc00066efc8 pc=0x37b2e46 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00066efe8 sp=0xc00066efe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).Start /go/src/github.com/milvus-io/milvus/internal/indexnode/task_scheduler.go:266 +0x6a

goroutine 305 [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0013e1978 sp=0xc0013e1958 pc=0x17870d6 runtime.netpollblock(0xc0013e19e8?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0013e19b0 sp=0xc0013e1978 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8fb38, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0013e19d0 sp=0xc0013e19b0 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc0010fd280?, 0xc001124880?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0013e19f8 sp=0xc0013e19d0 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc0010fd280) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0013e1aa0 sp=0xc0013e19f8 pc=0x18471fd net.(*netFD).accept(0xc0010fd280) /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc0013e1b58 sp=0xc0013e1aa0 pc=0x1a6bc15 net.(*TCPListener).accept(0xc00054eed0) /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc0013e1b80 sp=0xc0013e1b58 pc=0x1a8a025 net.(*TCPListener).Accept(0xc00054eed0) /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc0013e1bb0 sp=0xc0013e1b80 pc=0x1a88fbd google.golang.org/grpc.(*Server).Serve(0xc001144000, {0x4bf8310?, 0xc00054eed0}) /go/pkg/mod/google.golang.org/[email protected]/server.go:824 +0x475 fp=0xc0013e1cf8 sp=0xc0013e1bb0 pc=0x1ea13d5 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).startGrpcLoop(0xc0001f34d0, 0x0?) /go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:257 +0xa0b fp=0xc0013e1fc0 sp=0xc0013e1cf8 pc=0x3c1332b github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init.func1() /go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:148 +0x2a fp=0xc0013e1fe0 sp=0xc0013e1fc0 pc=0x3c128ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0013e1fe8 sp=0xc0013e1fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init /go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:148 +0x696

goroutine 323 [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000dc7768 sp=0xc000dc7748 pc=0x17870d6 runtime.netpollblock(0xc000dc77d8?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000dc77a0 sp=0xc000dc7768 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8fa48, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000dc77c0 sp=0xc000dc77a0 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc001202400?, 0xc000658540?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000dc77e8 sp=0xc000dc77c0 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc001202400) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000dc7890 sp=0xc000dc77e8 pc=0x18471fd net.(*netFD).accept(0xc001202400) /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000dc7948 sp=0xc000dc7890 pc=0x1a6bc15 net.(*TCPListener).accept(0xc0001568d0) /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000dc7970 sp=0xc000dc7948 pc=0x1a8a025 net.(*TCPListener).Accept(0xc0001568d0) /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000dc79a0 sp=0xc000dc7970 pc=0x1a88fbd google.golang.org/grpc.(*Server).Serve(0xc00122a000, {0x4bf8310?, 0xc0001568d0}) /go/pkg/mod/google.golang.org/[email protected]/server.go:824 +0x475 fp=0xc000dc7ae8 sp=0xc000dc79a0 pc=0x1ea13d5 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startInternalGrpc(0xc00044e000, 0x4c49, 0xc000440c00?) /go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:355 +0xd59 fp=0xc000dc7fb8 sp=0xc000dc7ae8 pc=0x3b6e9b9 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startInternalRPCServer.func1() /go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:196 +0x2e fp=0xc000dc7fe0 sp=0xc000dc7fb8 pc=0x3b6be2e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000dc7fe8 sp=0xc000dc7fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startInternalRPCServer /go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:196 +0xa5

goroutine 433 [select]: runtime.gopark(0xc0013e3f10?, 0x5?, 0x2?, 0x0?, 0xc0013e38ae?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0013e36e8 sp=0xc0013e36c8 pc=0x17870d6 runtime.selectgo(0xc0013e3f10, 0xc0013e38a4, 0xc001000840?, 0x0, 0x3faa6e0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0013e3828 sp=0xc0013e36e8 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*watchGrpcStream).run(0xc00123f110) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:537 +0x354 fp=0xc0013e3fc8 sp=0xc0013e3828 pc=0x1f6f1b4 go.etcd.io/etcd/client/v3.(*watcher).newWatcherGrpcStream.func1() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:285 +0x26 fp=0xc0013e3fe0 sp=0xc0013e3fc8 pc=0x1f6daa6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0013e3fe8 sp=0xc0013e3fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watcher).newWatcherGrpcStream /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:285 +0x34d

goroutine 452 [IO wait]: runtime.gopark(0xc001279378?, 0xb?, 0x0?, 0x0?, 0x11?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000675ad8 sp=0xc000675ab8 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000675b10 sp=0xc000675ad8 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8f868, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000675b30 sp=0xc000675b10 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc00148e280?, 0xc001242000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000675b58 sp=0xc000675b30 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00148e280, {0xc001242000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000675bf0 sp=0xc000675b58 pc=0x1842cd9 net.(*netFD).Read(0xc00148e280, {0xc001242000?, 0x3?, 0xc0004751e0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000675c38 sp=0xc000675bf0 pc=0x1a69a29 net.(*conn).Read(0xc000446440, {0xc001242000?, 0x174e9e0?, 0xc000cdc000?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000675c80 sp=0xc000675c38 pc=0x1a7ea05 net.(*TCPConn).Read(0xc000cdc000?, {0xc001242000?, 0x7fcba69aaf00?, 0x18?}) :1 +0x29 fp=0xc000675cb0 sp=0xc000675c80 pc=0x1a95569 bufio.(*Reader).Read(0xc000441380, {0xc00046f1c0, 0x9, 0x10000001d?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000675ce8 sp=0xc000675cb0 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc000441380}, {0xc00046f1c0, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000675d30 sp=0xc000675ce8 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc00046f1c0?, 0x9?, 0xc00044d290?}, {0x4bd2e40?, 0xc000441380?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc000675d80 sp=0xc000675d30 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc00046f180) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc000675e30 sp=0xc000675d80 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00122c240, 0xc0004411a0?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x257 fp=0xc000675fc0 sp=0xc000675e30 pc=0x1e19d37 google.golang.org/grpc/internal/transport.newHTTP2Client.func11() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2a fp=0xc000675fe0 sp=0xc000675fc0 pc=0x1e0f5ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000675fe8 sp=0xc000675fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1e2a

goroutine 371 [select]: runtime.gopark(0xc00123b5b0?, 0x2?, 0x10?, 0x0?, 0xc00123b59c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00123b428 sp=0xc00123b408 pc=0x17870d6 runtime.selectgo(0xc00123b5b0, 0xc00123b598, 0x1dab7a9?, 0x0, 0xc000272fc0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00123b568 sp=0xc00123b428 pc=0x1797bfe github.com/soheilhy/cmux.muxListener.Accept(...) /go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:262 github.com/soheilhy/cmux.(*muxListener).Accept(0x4c30f20?) :1 +0x7e fp=0xc00123b5e0 sp=0xc00123b568 pc=0x23646be google.golang.org/grpc.(*Server).Serve(0xc00122a1e0, {0x4bf9060?, 0xc0011248a0}) /go/pkg/mod/google.golang.org/[email protected]/server.go:824 +0x475 fp=0xc00123b728 sp=0xc00123b5e0 pc=0x1ea13d5 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc(0xc00044e000, 0x4c4a, 0xc000440c00?) /go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:291 +0x17d7 fp=0xc00123bfb8 sp=0xc00123b728 pc=0x3b6d777 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalRPCServer.func1() /go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:201 +0x2e fp=0xc00123bfe0 sp=0xc00123bfb8 pc=0x3b6bf6e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00123bfe8 sp=0xc00123bfe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalRPCServer /go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:201 +0xa5

goroutine 216 [select]: runtime.gopark(0xc00115efc0?, 0x2?, 0xc0?, 0x83?, 0xc00115ef6c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00115edc8 sp=0xc00115eda8 pc=0x17870d6 runtime.selectgo(0xc00115efc0, 0xc00115ef68, 0x4bd3f60?, 0x0, 0xb?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00115ef08 sp=0xc00115edc8 pc=0x1797bfe github.com/milvus-io/milvus/pkg/util/retry.Do({0x4bfad98, 0xc000f3e190}, 0xc00115f298, {0xc00115f188, 0x3, 0xc00115f510?}) /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:56 +0x4f1 fp=0xc00115f0b0 sp=0xc00115ef08 pc=0x2655a51 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call(0x4c4bec0, {0x4bfad98, 0xc000f3e190}, 0xc000ffca20) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411 +0x5b3 fp=0xc00115f300 sp=0xc00115f0b0 pc=0x346c553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call(0x4c4bec0, {0x4bfadd0, 0xc00005e040}, 0xc0000c4801) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489 +0x96 fp=0xc00115f490 sp=0xc00115f300 pc=0x346bb76 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall(...) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall({0x4bfadd0?, 0xc00005e040?}, 0x42779c0?) /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:503 +0x3f fp=0xc00115f4c8 sp=0xc00115f490 pc=0x346f2df github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]({0x4bfadd0, 0xc00005e040?}, 0xc0001cf320, 0xc0001cf350) /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 +0xc2 fp=0xc00115f500 sp=0xc00115f4c8 pc=0x3471782 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates(0xc00033c740?, {0x4bfadd0?, 0xc00005e040}, 0x1756da7?, {0x20?, 0x4347280?, 0x1?}) /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:121 +0x85 fp=0xc00115f548 sp=0xc00115f500 pc=0x34613e5 github.com/milvus-io/milvus/internal/types.RootCoordClient.GetComponentStates({0x4c4d978?, 0xc0001cf320?}, {0x4bfadd0?, 0xc00005e040?}, 0x456a420?, {0x0?, 0x17569ca?, 0x7fcba69ab3c8?}) :1 +0x62 fp=0xc00115f590 sp=0xc00115f548 pc=0x3419022 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1() /go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:37 +0xca fp=0xc00115f658 sp=0xc00115f590 pc=0x3796f6a github.com/milvus-io/milvus/pkg/util/retry.Do({0x4bfadd0, 0xc00005e040}, 0xc00115f860, {0xc00115f850, 0x2, 0xc0002ea990?}) /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:44 +0x148 fp=0xc00115f800 sp=0xc00115f658 pc=0x26556a8 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]({0x4bfadd0?, 0xc00005e040?}, {0x4c4d978?, 0xc0001cf320?}, {0x46b8c68?, 0x18?}, {0xc00115f92c, 0x1, 0x1}, 0xc00053e220?, ...) /go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:61 +0x145 fp=0xc00115f8c8 sp=0xc00115f800 pc=0x37973c5 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy... /go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:82 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init(0xc0002e7e00) /go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:262 +0x745 fp=0xc00115fe48 sp=0xc00115f8c8 pc=0x3795345 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run(0x17bdae7?) /go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:188 +0x25 fp=0xc00115fe88 sp=0xc00115fe48 pc=0x3794865 github.com/milvus-io/milvus/cmd/components.(*DataNode).Run(0xc00044a6c0?) /go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:53 +0x2f fp=0xc00115ff58 sp=0xc00115fe88 pc=0x3d3b02f github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113 +0x15a fp=0xc00115ffe0 sp=0xc00115ff58 pc=0x3d4157a runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00115ffe8 sp=0xc00115ffe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:100 +0x158

goroutine 552 [chan receive]: runtime.gopark(0x412d040?, 0x1764c00?, 0x28?, 0xbf?, 0xc00065bed0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00065be88 sp=0xc00065be68 pc=0x17870d6 runtime.chanrecv(0xc0016704e0, 0xc00065bf90, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00065bf18 sp=0xc00065be88 pc=0x174f5bd runtime.chanrecv2(0x412d040?, 0xc001288030?) /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00065bf40 sp=0xc00065bf18 pc=0x174f0f8 github.com/milvus-io/milvus/pkg/util/cache.(*localCache[...]).processEntries(0x4c538c0) /go/src/github.com/milvus-io/milvus/pkg/util/cache/local_cache.go:281 +0x9f fp=0xc00065bfc0 sp=0xc00065bf40 pc=0x3014cdf github.com/milvus-io/milvus/pkg/util/cache.(*localCache[...]).init.func1() /go/src/github.com/milvus-io/milvus/pkg/util/cache/local_cache.go:108 +0x2a fp=0xc00065bfe0 sp=0xc00065bfc0 pc=0x301324a runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00065bfe8 sp=0xc00065bfe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/pkg/util/cache.(*localCache[...]).init /go/src/github.com/milvus-io/milvus/pkg/util/cache/local_cache.go:108 +0x1dc

goroutine 574 [select]: runtime.gopark(0xc001e37f08?, 0x2?, 0xa3?, 0x50?, 0xc001e37edc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001e37d68 sp=0xc001e37d48 pc=0x17870d6 runtime.selectgo(0xc001e37f08, 0xc001e37ed8, 0x1a95869?, 0x0, 0xc001e1c000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001e37ea8 sp=0xc001e37d68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0019c1590, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc001e37f38 sp=0xc001e37ea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000f9d650) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc001e37f98 sp=0xc001e37f38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:336 +0xce fp=0xc001e37fe0 sp=0xc001e37f98 pc=0x1e1d26e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001e37fe8 sp=0xc001e37fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.NewServerTransport /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:333 +0x1ab3

goroutine 396 [IO wait]: runtime.gopark(0xc000431cd8?, 0xb?, 0x0?, 0x0?, 0xb?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000672ad8 sp=0xc000672ab8 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000672b10 sp=0xc000672ad8 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8f688, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000672b30 sp=0xc000672b10 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc0012b6080?, 0xc0012ba000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000672b58 sp=0xc000672b30 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0012b6080, {0xc0012ba000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000672bf0 sp=0xc000672b58 pc=0x1842cd9 net.(*netFD).Read(0xc0012b6080, {0xc0012ba000?, 0x1060100000000?, 0x8?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000672c38 sp=0xc000672bf0 pc=0x1a69a29 net.(*conn).Read(0xc00128a018, {0xc0012ba000?, 0xc000335740?, 0xc0014503c0?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000672c80 sp=0xc000672c38 pc=0x1a7ea05 net.(*TCPConn).Read(0x801010601?, {0xc0012ba000?, 0xc000672d28?, 0x801010601?}) :1 +0x29 fp=0xc000672cb0 sp=0xc000672c80 pc=0x1a95569 bufio.(*Reader).Read(0xc00129a1e0, {0xc00140e2e0, 0x9, 0x7fcba69ab878?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000672ce8 sp=0xc000672cb0 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc00129a1e0}, {0xc00140e2e0, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000672d30 sp=0xc000672ce8 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc00140e2e0?, 0x9?, 0x4f8910642?}, {0x4bd2e40?, 0xc00129a1e0?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc000672d80 sp=0xc000672d30 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc00140e2a0) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc000672e30 sp=0xc000672d80 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00147e000, 0x0?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x257 fp=0xc000672fc0 sp=0xc000672e30 pc=0x1e19d37 google.golang.org/grpc/internal/transport.newHTTP2Client.func11() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2a fp=0xc000672fe0 sp=0xc000672fc0 pc=0x1e0f5ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000672fe8 sp=0xc000672fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1e2a

goroutine 389 [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000671980 sp=0xc000671960 pc=0x17870d6 runtime.netpollblock(0xc0006719f0?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0006719b8 sp=0xc000671980 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8f958, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0006719d8 sp=0xc0006719b8 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc00141a200?, 0xc0014144e0?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000671a00 sp=0xc0006719d8 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc00141a200) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000671aa8 sp=0xc000671a00 pc=0x18471fd net.(*netFD).accept(0xc00141a200) /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000671b60 sp=0xc000671aa8 pc=0x1a6bc15 net.(*TCPListener).accept(0xc00140a180) /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000671b88 sp=0xc000671b60 pc=0x1a8a025 net.(*TCPListener).Accept(0xc00140a180) /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000671bb8 sp=0xc000671b88 pc=0x1a88fbd google.golang.org/grpc.(*Server).Serve(0xc001458000, {0x4bf8310?, 0xc00140a180}) /go/pkg/mod/google.golang.org/[email protected]/server.go:824 +0x475 fp=0xc000671d00 sp=0xc000671bb8 pc=0x1ea13d5 github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop(0xc000e75ce0, 0x5283) /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:215 +0x9bf fp=0xc000671fc0 sp=0xc000671d00 pc=0x3c9b4bf github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init.func1() /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:119 +0x2a fp=0xc000671fe0 sp=0xc000671fc0 pc=0x3c9a7aa runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000671fe8 sp=0xc000671fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:119 +0x6c5

goroutine 391 [select]: runtime.gopark(0xc001401f98?, 0x2?, 0x2?, 0x0?, 0xc001401f7c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001401e00 sp=0xc001401de0 pc=0x17870d6 runtime.selectgo(0xc001401f98, 0xc001401f78, 0xc00140a2a0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001401f40 sp=0xc001401e00 pc=0x1797bfe google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001416440) /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x73 fp=0xc001401fc8 sp=0xc001401f40 pc=0x1e8c193 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x26 fp=0xc001401fe0 sp=0xc001401fc8 pc=0x1e8c0e6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001401fe8 sp=0xc001401fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x22a

goroutine 412 [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001237930 sp=0xc001237910 pc=0x17870d6 runtime.netpollblock(0xc0012379a0?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc001237968 sp=0xc001237930 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8f598, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc001237988 sp=0xc001237968 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc00148e200?, 0xc000658940?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0012379b0 sp=0xc001237988 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc00148e200) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc001237a58 sp=0xc0012379b0 pc=0x18471fd net.(*netFD).accept(0xc00148e200) /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc001237b10 sp=0xc001237a58 pc=0x1a6bc15 net.(*TCPListener).accept(0xc000662be8) /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc001237b38 sp=0xc001237b10 pc=0x1a8a025 net.(*TCPListener).Accept(0xc000662be8) /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc001237b68 sp=0xc001237b38 pc=0x1a88fbd google.golang.org/grpc.(*Server).Serve(0xc0014a2000, {0x4bf8310?, 0xc000662be8}) /go/pkg/mod/google.golang.org/[email protected]/server.go:824 +0x475 fp=0xc001237cb0 sp=0xc001237b68 pc=0x1ea13d5 github.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).startGrpcLoop(0xc0006556e0, 0x5281) /go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:138 +0xaed fp=0xc001237fc0 sp=0xc001237cb0 pc=0x37b9f0d github.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init.func2() /go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:163 +0x2a fp=0xc001237fe0 sp=0xc001237fc0 pc=0x37bad2a runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001237fe8 sp=0xc001237fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init /go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:163 +0x29d

goroutine 267 [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000465998 sp=0xc000465978 pc=0x17870d6 runtime.netpollblock(0xc000465a08?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0004659d0 sp=0xc000465998 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8f2c8, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0004659f0 sp=0xc0004659d0 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc0012b6380?, 0xc0005ea340?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000465a18 sp=0xc0004659f0 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc0012b6380) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000465ac0 sp=0xc000465a18 pc=0x18471fd net.(*netFD).accept(0xc0012b6380) /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000465b78 sp=0xc000465ac0 pc=0x1a6bc15 net.(*TCPListener).accept(0xc001288120) /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000465ba0 sp=0xc000465b78 pc=0x1a8a025 net.(*TCPListener).Accept(0xc001288120) /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000465bd0 sp=0xc000465ba0 pc=0x1a88fbd google.golang.org/grpc.(*Server).Serve(0xc0012d0000, {0x4bf8310?, 0xc001288120}) /go/pkg/mod/google.golang.org/[email protected]/server.go:824 +0x475 fp=0xc000465d18 sp=0xc000465bd0 pc=0x1ea13d5 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).startGrpcLoop(0xc0002e7e00, 0xc000690001?) /go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:168 +0x97f fp=0xc000465fc0 sp=0xc000465d18 pc=0x379429f github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).startGrpc.func1() /go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:101 +0x2a fp=0xc000465fe0 sp=0xc000465fc0 pc=0x37938ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000465fe8 sp=0xc000465fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).startGrpc /go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:101 +0x89

goroutine 397 [select]: runtime.gopark(0xc000e19f08?, 0x2?, 0xa3?, 0x50?, 0xc000e19edc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000e19d68 sp=0xc000e19d48 pc=0x17870d6 runtime.selectgo(0xc000e19f08, 0xc000e19ed8, 0x1a95869?, 0x0, 0xc00146e000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000e19ea8 sp=0xc000e19d68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00141cbe0, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc000e19f38 sp=0xc000e19ea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001680000) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc000e19f98 sp=0xc000e19f38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x7b fp=0xc000e19fe0 sp=0xc000e19f98 pc=0x1e0f4bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000e19fe8 sp=0xc000e19fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x23fc

goroutine 394 [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001467978 sp=0xc001467958 pc=0x17870d6 runtime.netpollblock(0xc0014679e8?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0014679b0 sp=0xc001467978 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8f4a8, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0014679d0 sp=0xc0014679b0 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc00141a300?, 0xc001414980?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0014679f8 sp=0xc0014679d0 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc00141a300) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc001467aa0 sp=0xc0014679f8 pc=0x18471fd net.(*netFD).accept(0xc00141a300) /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc001467b58 sp=0xc001467aa0 pc=0x1a6bc15 net.(*TCPListener).accept(0xc00140a2d0) /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc001467b80 sp=0xc001467b58 pc=0x1a8a025 net.(*TCPListener).Accept(0xc00140a2d0) /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc001467bb0 sp=0xc001467b80 pc=0x1a88fbd google.golang.org/grpc.(*Server).Serve(0xc0014581e0, {0x4bf8310?, 0xc00140a2d0}) /go/pkg/mod/google.golang.org/[email protected]/server.go:824 +0x475 fp=0xc001467cf8 sp=0xc001467bb0 pc=0x1ea13d5 github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).startGrpcLoop(0xc000654de0, 0x3415) /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:199 +0xa4b fp=0xc001467fc0 sp=0xc001467cf8 pc=0x356cf8b github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).startGrpc.func1() /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:136 +0x2a fp=0xc001467fe0 sp=0xc001467fc0 pc=0x356c50a runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001467fe8 sp=0xc001467fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).startGrpc /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:136 +0x89

goroutine 426 [select]: runtime.gopark(0xc000dc5f98?, 0x2?, 0x2?, 0x0?, 0xc000dc5f7c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000dc5e00 sp=0xc000dc5de0 pc=0x17870d6 runtime.selectgo(0xc000dc5f98, 0xc000dc5f78, 0xc000156540?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000dc5f40 sp=0xc000dc5e00 pc=0x1797bfe google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00046cd00) /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x73 fp=0xc000dc5fc8 sp=0xc000dc5f40 pc=0x1e8c193 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x26 fp=0xc000dc5fe0 sp=0xc000dc5fc8 pc=0x1e8c0e6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000dc5fe8 sp=0xc000dc5fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x22a

goroutine 625 [select]: runtime.gopark(0xc001e11f08?, 0x2?, 0xa3?, 0x50?, 0xc001e11edc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001e11d68 sp=0xc001e11d48 pc=0x17870d6 runtime.selectgo(0xc001e11f08, 0xc001e11ed8, 0x1a95869?, 0x0, 0xc001e8a000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001e11ea8 sp=0xc001e11d68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001b70b40, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc001e11f38 sp=0xc001e11ea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000f9d570) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc001e11f98 sp=0xc001e11f38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:336 +0xce fp=0xc001e11fe0 sp=0xc001e11f98 pc=0x1e1d26e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001e11fe8 sp=0xc001e11fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.NewServerTransport /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:333 +0x1ab3

goroutine 217 [select]: runtime.gopark(0xc001405f98?, 0x2?, 0x2?, 0x0?, 0xc001405f7c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001405e00 sp=0xc001405de0 pc=0x17870d6 runtime.selectgo(0xc001405f98, 0xc001405f78, 0xc000ede720?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001405f40 sp=0xc001405e00 pc=0x1797bfe google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000f3aac0) /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x73 fp=0xc001405fc8 sp=0xc001405f40 pc=0x1e8c193 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x26 fp=0xc001405fe0 sp=0xc001405fc8 pc=0x1e8c0e6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001405fe8 sp=0xc001405fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x22a

goroutine 432 [select]: runtime.gopark(0xc0016727b0?, 0x2?, 0xb8?, 0x27?, 0xc001672784?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001672600 sp=0xc0016725e0 pc=0x17870d6 runtime.selectgo(0xc0016727b0, 0xc001672780, 0x4bbc788?, 0x0, 0xc0013fe750?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001672740 sp=0xc001672600 pc=0x1797bfe context.propagateCancel.func1() /usr/local/go/src/context/context.go:333 +0xa5 fp=0xc0016727e0 sp=0xc001672740 pc=0x1870f65 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0016727e8 sp=0xc0016727e0 pc=0x17bd9a1 created by context.propagateCancel /usr/local/go/src/context/context.go:332 +0x1bc

goroutine 468 [select]: runtime.gopark(0xc0014c5f08?, 0x2?, 0xa3?, 0x50?, 0xc0014c5edc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0014c5d68 sp=0xc0014c5d48 pc=0x17870d6 runtime.selectgo(0xc0014c5f08, 0xc0014c5ed8, 0x1a95869?, 0x0, 0xc001628000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0014c5ea8 sp=0xc0014c5d68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0006560a0, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc0014c5f38 sp=0xc0014c5ea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0014aa070) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc0014c5f98 sp=0xc0014c5f38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x7b fp=0xc0014c5fe0 sp=0xc0014c5f98 pc=0x1e0f4bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0014c5fe8 sp=0xc0014c5fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x23fc

goroutine 467 [IO wait]: runtime.gopark(0xc001af5438?, 0xb?, 0x0?, 0x0?, 0x17?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000d77ad8 sp=0xc000d77ab8 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000d77b10 sp=0xc000d77ad8 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8f0e8, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000d77b30 sp=0xc000d77b10 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc00035a200?, 0xc001620000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000d77b58 sp=0xc000d77b30 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00035a200, {0xc001620000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000d77bf0 sp=0xc000d77b58 pc=0x1842cd9 net.(*netFD).Read(0xc00035a200, {0xc001620000?, 0x1060100000000?, 0x8?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000d77c38 sp=0xc000d77bf0 pc=0x1a69a29 net.(*conn).Read(0xc000668008, {0xc001620000?, 0xc0003347c0?, 0xc0014981e0?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000d77c80 sp=0xc000d77c38 pc=0x1a7ea05 net.(*TCPConn).Read(0x801010601?, {0xc001620000?, 0xc000d77d28?, 0x801010601?}) :1 +0x29 fp=0xc000d77cb0 sp=0xc000d77c80 pc=0x1a95569 bufio.(*Reader).Read(0xc000440000, {0xc00140e040, 0x9, 0x7fcba69ab878?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000d77ce8 sp=0xc000d77cb0 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc000440000}, {0xc00140e040, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000d77d30 sp=0xc000d77ce8 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc00140e040?, 0x9?, 0x300ca8696?}, {0x4bd2e40?, 0xc000440000?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc000d77d80 sp=0xc000d77d30 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc00140e000) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc000d77e30 sp=0xc000d77d80 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0014b4240, 0x0?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x257 fp=0xc000d77fc0 sp=0xc000d77e30 pc=0x1e19d37 google.golang.org/grpc/internal/transport.newHTTP2Client.func11() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2a fp=0xc000d77fe0 sp=0xc000d77fc0 pc=0x1e0f5ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000d77fe8 sp=0xc000d77fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1e2a

goroutine 757 [select]: runtime.gopark(0xc001d6df08?, 0x2?, 0xa3?, 0x50?, 0xc001d6dedc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001d6dd68 sp=0xc001d6dd48 pc=0x17870d6 runtime.selectgo(0xc001d6df08, 0xc001d6ded8, 0x1a95869?, 0x0, 0xc001d54000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001d6dea8 sp=0xc001d6dd68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0016ab450, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc001d6df38 sp=0xc001d6dea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00179f570) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc001d6df98 sp=0xc001d6df38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x7b fp=0xc001d6dfe0 sp=0xc001d6df98 pc=0x1e0f4bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001d6dfe8 sp=0xc001d6dfe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x23fc

goroutine 439 [IO wait]: runtime.gopark(0xc00129cea0?, 0xb?, 0x0?, 0x0?, 0x14?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000fa6ad8 sp=0xc000fa6ab8 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000fa6b10 sp=0xc000fa6ad8 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca9de8eff8, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000fa6b30 sp=0xc000fa6b10 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc0015ea000?, 0xc001778000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000fa6b58 sp=0xc000fa6b30 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0015ea000, {0xc001778000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000fa6bf0 sp=0xc000fa6b58 pc=0x1842cd9 net.(*netFD).Read(0xc0015ea000, {0xc001778000?, 0x1060100000000?, 0x8?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000fa6c38 sp=0xc000fa6bf0 pc=0x1a69a29 net.(*conn).Read(0xc0015f0018, {0xc001778000?, 0x174e9e0?, 0xc001671140?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000fa6c80 sp=0xc000fa6c38 pc=0x1a7ea05 net.(*TCPConn).Read(0x801010601?, {0xc001778000?, 0xc000fa6d28?, 0x801010601?}) :1 +0x29 fp=0xc000fa6cb0 sp=0xc000fa6c80 pc=0x1a95569 bufio.(*Reader).Read(0xc0015e02a0, {0xc001790040, 0x9, 0x7fcba69ab878?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000fa6ce8 sp=0xc000fa6cb0 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc0015e02a0}, {0xc001790040, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000fa6d30 sp=0xc000fa6ce8 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc001790040?, 0x9?, 0x4bc5bfbc2?}, {0x4bd2e40?, 0xc0015e02a0?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc000fa6d80 sp=0xc000fa6d30 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc001790000) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc000fa6e30 sp=0xc000fa6d80 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001794000, 0xc000277340?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x257 fp=0xc000fa6fc0 sp=0xc000fa6e30 pc=0x1e19d37 google.golang.org/grpc/internal/transport.newHTTP2Client.func11() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2a fp=0xc000fa6fe0 sp=0xc000fa6fc0 pc=0x1e0f5ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000fa6fe8 sp=0xc000fa6fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1e2a

goroutine 440 [select]: runtime.gopark(0xc0017a7f08?, 0x2?, 0xa3?, 0x50?, 0xc0017a7edc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0017a7d68 sp=0xc0017a7d48 pc=0x17870d6 runtime.selectgo(0xc0017a7f08, 0xc0017a7ed8, 0x1a95869?, 0x0, 0xc001780000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0017a7ea8 sp=0xc0017a7d68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0013fc3c0, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc0017a7f38 sp=0xc0017a7ea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00179e000) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc0017a7f98 sp=0xc0017a7f38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x7b fp=0xc0017a7fe0 sp=0xc0017a7f98 pc=0x1e0f4bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0017a7fe8 sp=0xc0017a7fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x23fc

goroutine 453 [runnable]: runtime.gopark(0xc0014b3f08?, 0x2?, 0xa3?, 0x50?, 0xc0014b3edc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0014b3d68 sp=0xc0014b3d48 pc=0x17870d6 runtime.selectgo(0xc0014b3f08, 0xc0014b3ed8, 0x1a95869?, 0x0, 0xc00124a000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0014b3ea8 sp=0xc0014b3d68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0012323c0, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc0014b3f38 sp=0xc0014b3ea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0014aa000) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc0014b3f98 sp=0xc0014b3f38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x7b fp=0xc0014b3fe0 sp=0xc0014b3f98 pc=0x1e0f4bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0014b3fe8 sp=0xc0014b3fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x23fc

goroutine 219 [select]: runtime.gopark(0xc0014bdf08?, 0x2?, 0xa3?, 0x50?, 0xc0014bdedc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0014bdd68 sp=0xc0014bdd48 pc=0x17870d6 runtime.selectgo(0xc0014bdf08, 0xc0014bded8, 0x1a95869?, 0x0, 0xc0012e4000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0014bdea8 sp=0xc0014bdd68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000657a90, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc0014bdf38 sp=0xc0014bdea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001680070) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc0014bdf98 sp=0xc0014bdf38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x7b fp=0xc0014bdfe0 sp=0xc0014bdf98 pc=0x1e0f4bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0014bdfe8 sp=0xc0014bdfe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x23fc

goroutine 464 [IO wait]: runtime.gopark(0xc00149dba0?, 0xb?, 0x0?, 0x0?, 0x21?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000673ad8 sp=0xc000673ab8 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000673b10 sp=0xc000673ad8 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca742ea3e0, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000673b30 sp=0xc000673b10 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc0015ea380?, 0xc000c68000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000673b58 sp=0xc000673b30 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc0015ea380, {0xc000c68000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000673bf0 sp=0xc000673b58 pc=0x1842cd9 net.(*netFD).Read(0xc0015ea380, {0xc000c68000?, 0x1060100000000?, 0x8?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000673c38 sp=0xc000673bf0 pc=0x1a69a29 net.(*conn).Read(0xc000556238, {0xc000c68000?, 0x174e9e0?, 0xc0004419e0?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000673c80 sp=0xc000673c38 pc=0x1a7ea05 net.(*TCPConn).Read(0x801010601?, {0xc000c68000?, 0xc000673d28?, 0x801010601?}) :1 +0x29 fp=0xc000673cb0 sp=0xc000673c80 pc=0x1a95569 bufio.(*Reader).Read(0xc001126900, {0xc000040580, 0x9, 0x7fcba69ab878?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000673ce8 sp=0xc000673cb0 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc001126900}, {0xc000040580, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000673d30 sp=0xc000673ce8 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc000040580?, 0x9?, 0x4ba7669c0?}, {0x4bd2e40?, 0xc001126900?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc000673d80 sp=0xc000673d30 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc000040540) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc000673e30 sp=0xc000673d80 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00122c900, 0xc0003365c0?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x257 fp=0xc000673fc0 sp=0xc000673e30 pc=0x1e19d37 google.golang.org/grpc/internal/transport.newHTTP2Client.func11() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2a fp=0xc000673fe0 sp=0xc000673fc0 pc=0x1e0f5ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000673fe8 sp=0xc000673fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1e2a

goroutine 756 [IO wait]: runtime.gopark(0x1b?, 0xb?, 0x0?, 0x0?, 0x1b?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0017abad8 sp=0xc0017abab8 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0017abb10 sp=0xc0017abad8 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca742ea4d0, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0017abb30 sp=0xc0017abb10 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc001140000?, 0xc001d4c000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0017abb58 sp=0xc0017abb30 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc001140000, {0xc001d4c000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0017abbf0 sp=0xc0017abb58 pc=0x1842cd9 net.(*netFD).Read(0xc001140000, {0xc001d4c000?, 0x1040100000000?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0017abc38 sp=0xc0017abbf0 pc=0x1a69a29 net.(*conn).Read(0xc00128a948, {0xc001d4c000?, 0x412d040?, 0xc0017abd68?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0017abc80 sp=0xc0017abc38 pc=0x1a7ea05 net.(*TCPConn).Read(0x1010401?, {0xc001d4c000?, 0x17569ca?, 0x1010401?}) :1 +0x29 fp=0xc0017abcb0 sp=0xc0017abc80 pc=0x1a95569 bufio.(*Reader).Read(0xc001d4a3c0, {0xc001d64040, 0x9, 0x30?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0017abce8 sp=0xc0017abcb0 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc001d4a3c0}, {0xc001d64040, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0017abd30 sp=0xc0017abce8 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc001d64040?, 0x9?, 0xc001b5ff80?}, {0x4bd2e40?, 0xc001d4a3c0?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc0017abd80 sp=0xc0017abd30 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc001d64000) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc0017abe30 sp=0xc0017abd80 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001d1e480, 0xc001164556?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x257 fp=0xc0017abfc0 sp=0xc0017abe30 pc=0x1e19d37 google.golang.org/grpc/internal/transport.newHTTP2Client.func11() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2a fp=0xc0017abfe0 sp=0xc0017abfc0 pc=0x1e0f5ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0017abfe8 sp=0xc0017abfe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1e2a

goroutine 638 [select]: runtime.gopark(0xc001ec5f08?, 0x2?, 0xa3?, 0x50?, 0xc001ec5edc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001ec5d68 sp=0xc001ec5d48 pc=0x17870d6 runtime.selectgo(0xc001ec5f08, 0xc001ec5ed8, 0x1a95869?, 0x0, 0xc001f80000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001ec5ea8 sp=0xc001ec5d68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0011201e0, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc001ec5f38 sp=0xc001ec5ea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000fd8310) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc001ec5f98 sp=0xc001ec5f38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:336 +0xce fp=0xc001ec5fe0 sp=0xc001ec5f98 pc=0x1e1d26e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001ec5fe8 sp=0xc001ec5fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.NewServerTransport /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:333 +0x1ab3

goroutine 581 [select]: runtime.gopark(0xc000fa5e40?, 0x5?, 0x88?, 0x5c?, 0xc000fa5de6?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000fa5c40 sp=0xc000fa5c20 pc=0x17870d6 runtime.selectgo(0xc000fa5e40, 0xc000fa5ddc, 0xc00065ef1c?, 0x1, 0x41cb860?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000fa5d80 sp=0xc000fa5c40 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*watchGrpcStream).serveSubstream(0xc00123f110, 0xc001000840, 0xc0014faa20) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:803 +0x265 fp=0xc000fa5fb8 sp=0xc000fa5d80 pc=0x1f71b65 go.etcd.io/etcd/client/v3.(*watchGrpcStream).run.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0x2e fp=0xc000fa5fe0 sp=0xc000fa5fb8 pc=0x1f70e0e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000fa5fe8 sp=0xc000fa5fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watchGrpcStream).run /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0xc2c

goroutine 575 [select]: runtime.gopark(0xc00065d770?, 0x4?, 0x0?, 0x0?, 0xc00065d6c0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00065d508 sp=0xc00065d4e8 pc=0x17870d6 runtime.selectgo(0xc00065d770, 0xc00065d6b8, 0x7fca742ea110?, 0x0, 0xe9?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00065d648 sp=0xc00065d508 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001e2e000) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:1150 +0x233 fp=0xc00065d7c8 sp=0xc00065d648 pc=0x1e24ab3 google.golang.org/grpc/internal/transport.NewServerTransport.func4() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:339 +0x26 fp=0xc00065d7e0 sp=0xc00065d7c8 pc=0x1e1d166 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00065d7e8 sp=0xc00065d7e0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.NewServerTransport /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:339 +0x1af8

goroutine 639 [select]: runtime.gopark(0xc0012a4770?, 0x4?, 0x9?, 0x0?, 0xc0012a46c0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012a4508 sp=0xc0012a44e8 pc=0x17870d6 runtime.selectgo(0xc0012a4770, 0xc0012a46b8, 0xc000582820?, 0x0, 0xc0012a46d0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012a4648 sp=0xc0012a4508 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00057c9c0) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:1150 +0x233 fp=0xc0012a47c8 sp=0xc0012a4648 pc=0x1e24ab3 google.golang.org/grpc/internal/transport.NewServerTransport.func4() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:339 +0x26 fp=0xc0012a47e0 sp=0xc0012a47c8 pc=0x1e1d166 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012a47e8 sp=0xc0012a47e0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.NewServerTransport /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:339 +0x1af8

goroutine 722 [select]: runtime.gopark(0xc00011d770?, 0x4?, 0x0?, 0x0?, 0xc00011d6c0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00011d508 sp=0xc00011d4e8 pc=0x17870d6 runtime.selectgo(0xc00011d770, 0xc00011d6b8, 0xc00011d6c0?, 0x0, 0x17569ca?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00011d648 sp=0xc00011d508 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001b249c0) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:1150 +0x233 fp=0xc00011d7c8 sp=0xc00011d648 pc=0x1e24ab3 google.golang.org/grpc/internal/transport.NewServerTransport.func4() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:339 +0x26 fp=0xc00011d7e0 sp=0xc00011d7c8 pc=0x1e1d166 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00011d7e8 sp=0xc00011d7e0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.NewServerTransport /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:339 +0x1af8

goroutine 582 [select]: runtime.gopark(0xc001293bf8?, 0x3?, 0xa?, 0x0?, 0xc001293b6a?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012939d0 sp=0xc0012939b0 pc=0x17870d6 runtime.selectgo(0xc001293bf8, 0xc001293b64, 0xc00141b400?, 0x0, 0xc000336140?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001293b10 sp=0xc0012939d0 pc=0x1797bfe github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).LivenessCheck.func1() /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:820 +0x215 fp=0xc001293fe0 sp=0xc001293b10 pc=0x3422a75 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001293fe8 sp=0xc001293fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).LivenessCheck /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:808 +0x22c

goroutine 376 [select]: runtime.gopark(0xc0012aff10?, 0x3?, 0x0?, 0xb?, 0xc0012afe6a?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012afcd8 sp=0xc0012afcb8 pc=0x17870d6 runtime.selectgo(0xc0012aff10, 0xc0012afe64, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012afe18 sp=0xc0012afcd8 pc=0x1797bfe github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.(*retentionInfo).retention(0xc00064d500) /go/src/github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_retention.go:90 +0x1e5 fp=0xc0012affc8 sp=0xc0012afe18 pc=0x34c82a5 github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.(*retentionInfo).startRetentionInfo.func1() /go/src/github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_retention.go:75 +0x26 fp=0xc0012affe0 sp=0xc0012affc8 pc=0x34c8086 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012affe8 sp=0xc0012affe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server.(*retentionInfo).startRetentionInfo /go/src/github.com/milvus-io/milvus/internal/mq/mqimpl/rocksmq/server/rocksmq_retention.go:75 +0x6a

goroutine 723 [IO wait]: runtime.gopark(0x7fffffffffffffff?, 0xb?, 0x0?, 0x0?, 0x19?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0017acaa0 sp=0xc0017aca80 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0017acad8 sp=0xc0017acaa0 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca742ea6b0, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0017acaf8 sp=0xc0017acad8 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc001b6e400?, 0xc001e82000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0017acb20 sp=0xc0017acaf8 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc001b6e400, {0xc001e82000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0017acbb8 sp=0xc0017acb20 pc=0x1842cd9 net.(*netFD).Read(0xc001b6e400, {0xc001e82000?, 0x1040100000000?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0017acc00 sp=0xc0017acbb8 pc=0x1a69a29 net.(*conn).Read(0xc0003347b0, {0xc001e82000?, 0x0?, 0x0?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0017acc48 sp=0xc0017acc00 pc=0x1a7ea05 net.(*TCPConn).Read(0x1010401?, {0xc001e82000?, 0x17569ca?, 0x1010401?}) :1 +0x29 fp=0xc0017acc78 sp=0xc0017acc48 pc=0x1a95569 bufio.(*Reader).Read(0xc001b6ca80, {0xc0005f99a0, 0x9, 0x30?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0017accb0 sp=0xc0017acc78 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc001b6ca80}, {0xc0005f99a0, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0017accf8 sp=0xc0017accb0 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc0005f99a0?, 0x9?, 0xc001e01d10?}, {0x4bd2e40?, 0xc001b6ca80?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc0017acd48 sp=0xc0017accf8 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005f9960) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc0017acdf8 sp=0xc0017acd48 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001b249c0, 0xc15abf53f08657a4?, 0x1000000000001?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:637 +0x174 fp=0xc0017acf08 sp=0xc0017acdf8 pc=0x1e20674 google.golang.org/grpc.(*Server).serveStreams(0xc000cc61e0, {0x4c10380?, 0xc001b249c0}) /go/pkg/mod/google.golang.org/[email protected]/server.go:950 +0x189 fp=0xc0017acf80 sp=0xc0017acf08 pc=0x1ea2529 google.golang.org/grpc.(*Server).handleRawConn.func1() /go/pkg/mod/google.golang.org/[email protected]/server.go:892 +0x46 fp=0xc0017acfe0 sp=0xc0017acf80 pc=0x1ea1da6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0017acfe8 sp=0xc0017acfe0 pc=0x17bd9a1 created by google.golang.org/grpc.(*Server).handleRawConn /go/pkg/mod/google.golang.org/[email protected]/server.go:891 +0x185

goroutine 580 [select]: runtime.gopark(0xc000468968?, 0x2?, 0xe5?, 0x36?, 0xc000468934?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0004687b8 sp=0xc000468798 pc=0x17870d6 runtime.selectgo(0xc000468968, 0xc000468930, 0x18aa0d1?, 0x0, 0xc0004f3b30?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0004688f8 sp=0xc0004687b8 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc001914870, {0xc000157c00, 0x5, 0x5}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:195 +0xa5 fp=0xc000468998 sp=0xc0004688f8 pc=0x1e29bc5 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc001914870, {0xc000157c00?, 0xc000cd85e8?, 0xc0004689e0?}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:175 +0x147 fp=0xc0004689d8 sp=0xc000468998 pc=0x1e29907 google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0014ff0e0, {0xc000157c00?, 0xc000468a58?, 0x1e10107?}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:514 +0x32 fp=0xc000468a20 sp=0xc0004689d8 pc=0x1e2afd2 io.ReadAtLeast({0x4bd8640, 0xc0014ff0e0}, {0xc000157c00, 0x5, 0x5}, 0x5) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000468a68 sp=0xc000468a20 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0014f9200, {0xc000157c00, 0x5, 0x5}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:498 +0xac fp=0xc000468aa8 sp=0xc000468a68 pc=0x1e2af4c google.golang.org/grpc.(*parser).recvMsg(0xc000157bf0, 0x7fffffff) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:597 +0x47 fp=0xc000468b28 sp=0xc000468aa8 pc=0x1e9d6c7 google.golang.org/grpc.recvAndDecompress(0xc000468c28?, 0xc0014f9200, {0x0, 0x0}, 0x7fffffff, 0x0, {0x0, 0x0}) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:729 +0x66 fp=0xc000468bc0 sp=0xc000468b28 pc=0x1e9df26 google.golang.org/grpc.recv(0xc00064c720?, {0x7fca735b3378, 0x6757a08}, 0x1756da7?, {0x0?, 0x0?}, {0x458dde0, 0xc001022cb0}, 0x1787452?, 0x0, ...) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:795 +0x6e fp=0xc000468c38 sp=0xc000468bc0 pc=0x1e9e60e google.golang.org/grpc.(*csAttempt).recvMsg(0xc00123f2b0, {0x458dde0?, 0xc001022cb0}, 0xc00057bd40?) /go/pkg/mod/google.golang.org/[email protected]/stream.go:1072 +0x2cc fp=0xc000468d68 sp=0xc000468c38 pc=0x1eb2d8c google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x174eb11?) /go/pkg/mod/google.golang.org/[email protected]/stream.go:909 +0x25 fp=0xc000468d98 sp=0xc000468d68 pc=0x1eb1f45 google.golang.org/grpc.(*clientStream).withRetry(0xc0014f8fc0, 0xc000468eb0, 0xc000468e80) /go/pkg/mod/google.golang.org/[email protected]/stream.go:744 +0x3be fp=0xc000468e08 sp=0xc000468d98 pc=0x1eb0b3e google.golang.org/grpc.(*clientStream).RecvMsg(0xc0014f8fc0, {0x458dde0?, 0xc001022cb0?}) /go/pkg/mod/google.golang.org/[email protected]/stream.go:908 +0x131 fp=0xc000468ee0 sp=0xc000468e08 pc=0x1eb1bb1 go.etcd.io/etcd/api/v3/etcdserverpb.(*watchWatchClient).Recv(0xc0004f3800) /go/pkg/mod/go.etcd.io/etcd/api/[email protected]/etcdserverpb/rpc.pb.go:6714 +0x4c fp=0xc000468f10 sp=0xc000468ee0 pc=0x1ee7f0c go.etcd.io/etcd/client/v3.(*watchGrpcStream).serveWatchClient(0xc00123f110, {0x4c0e468, 0xc0004f3800}) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:757 +0x6a fp=0xc000468fb8 sp=0xc000468f10 pc=0x1f717ca go.etcd.io/etcd/client/v3.(*watchGrpcStream).newWatchClient.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:912 +0x2e fp=0xc000468fe0 sp=0xc000468fb8 pc=0x1f7266e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000468fe8 sp=0xc000468fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watchGrpcStream).newWatchClient /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:912 +0x5eb

goroutine 508 [select]: runtime.gopark(0xc00011bf60?, 0x3?, 0xca?, 0x69?, 0xc00011bf1a?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00011bd68 sp=0xc00011bd48 pc=0x17870d6 runtime.selectgo(0xc00011bf60, 0xc00011bf14, 0x0?, 0x0, 0x4587480?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00011bea8 sp=0xc00011bd68 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*lessor).keepAliveCtxCloser(0xc00047c5a0, {0x4bfad98, 0xc00013acd0}, 0xc000d423c0?, 0xc00066b800) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:330 +0xd9 fp=0xc00011bfa8 sp=0xc00011bea8 pc=0x1f60ab9 go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:297 +0x36 fp=0xc00011bfe0 sp=0xc00011bfa8 pc=0x1f607d6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*lessor).KeepAlive /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:297 +0x5aa

goroutine 465 [select]: runtime.gopark(0xc000d39f08?, 0x2?, 0xa3?, 0x50?, 0xc000d39edc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000d39d68 sp=0xc000d39d48 pc=0x17870d6 runtime.selectgo(0xc000d39f08, 0xc000d39ed8, 0x1a95869?, 0x0, 0xc001010000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000d39ea8 sp=0xc000d39d68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001123810, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc000d39f38 sp=0xc000d39ea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00161e1c0) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc000d39f98 sp=0xc000d39f38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x7b fp=0xc000d39fe0 sp=0xc000d39f98 pc=0x1e0f4bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000d39fe8 sp=0xc000d39fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x23fc

goroutine 576 [IO wait]: runtime.gopark(0x7fffffffffffffff?, 0xb?, 0x0?, 0x0?, 0x20?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000ceaa0 sp=0xc0000cea80 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0000cead8 sp=0xc0000ceaa0 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca742ea110, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0000ceaf8 sp=0xc0000cead8 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc001140080?, 0xc001e14000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0000ceb20 sp=0xc0000ceaf8 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc001140080, {0xc001e14000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0000cebb8 sp=0xc0000ceb20 pc=0x1842cd9 net.(*netFD).Read(0xc001140080, {0xc001e14000?, 0x1040100000000?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0000cec00 sp=0xc0000cebb8 pc=0x1a69a29 net.(*conn).Read(0xc001428538, {0xc001e14000?, 0x0?, 0x0?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0000cec48 sp=0xc0000cec00 pc=0x1a7ea05 net.(*TCPConn).Read(0x10401?, {0xc001e14000?, 0x17569ca?, 0x10401?}) :1 +0x29 fp=0xc0000cec78 sp=0xc0000cec48 pc=0x1a95569 bufio.(*Reader).Read(0xc0019bb0e0, {0xc00140f620, 0x9, 0x30?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0000cecb0 sp=0xc0000cec78 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc0019bb0e0}, {0xc00140f620, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0000cecf8 sp=0xc0000cecb0 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc00140f620?, 0x9?, 0xc00057f470?}, {0x4bd2e40?, 0xc0019bb0e0?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc0000ced48 sp=0xc0000cecf8 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc00140f5e0) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc0000cedf8 sp=0xc0000ced48 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001e2e000, 0x10?, 0x10?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:637 +0x174 fp=0xc0000cef08 sp=0xc0000cedf8 pc=0x1e20674 google.golang.org/grpc.(*Server).serveStreams(0xc000cc61e0, {0x4c10380?, 0xc001e2e000}) /go/pkg/mod/google.golang.org/[email protected]/server.go:950 +0x189 fp=0xc0000cef80 sp=0xc0000cef08 pc=0x1ea2529 google.golang.org/grpc.(*Server).handleRawConn.func1() /go/pkg/mod/google.golang.org/[email protected]/server.go:892 +0x46 fp=0xc0000cefe0 sp=0xc0000cef80 pc=0x1ea1da6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000cefe8 sp=0xc0000cefe0 pc=0x17bd9a1 created by google.golang.org/grpc.(*Server).handleRawConn /go/pkg/mod/google.golang.org/[email protected]/server.go:891 +0x185

goroutine 480 [select]: runtime.gopark(0xc0013e7f08?, 0x4?, 0x5?, 0x0?, 0xc0013e7ed8?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0013e7d18 sp=0xc0013e7cf8 pc=0x17870d6 runtime.selectgo(0xc0013e7f08, 0xc0013e7ed0, 0x670cd80?, 0x0, 0x531d364d5?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0013e7e58 sp=0xc0013e7d18 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*lessor).sendKeepAliveLoop(0xc00047c5a0, {0x4c0e360, 0xc000f33720}) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:588 +0x1e9 fp=0xc0013e7fb8 sp=0xc0013e7e58 pc=0x1f62d69 go.etcd.io/etcd/client/v3.(*lessor).resetRecv.func3() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:498 +0x2e fp=0xc0013e7fe0 sp=0xc0013e7fb8 pc=0x1f6228e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0013e7fe8 sp=0xc0013e7fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*lessor).resetRecv /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:498 +0x2cd

goroutine 728 [select]: runtime.gopark(0xc00011c770?, 0x2?, 0xc0?, 0x7b?, 0xc00011c71c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00011c578 sp=0xc00011c558 pc=0x17870d6 runtime.selectgo(0xc00011c770, 0xc00011c718, 0xc00011c6f8?, 0x0, 0xc001b6cd20?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00011c6b8 sp=0xc00011c578 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001b60b40) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1666 +0x165 fp=0xc00011c7c8 sp=0xc00011c6b8 pc=0x1e1a525 google.golang.org/grpc/internal/transport.newHTTP2Client.func10() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:384 +0x26 fp=0xc00011c7e0 sp=0xc00011c7c8 pc=0x1e0f646 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00011c7e8 sp=0xc00011c7e0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:384 +0x1dac

goroutine 729 [IO wait]: runtime.gopark(0x1a?, 0xb?, 0x0?, 0x0?, 0x1a?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0017a9ad8 sp=0xc0017a9ab8 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0017a9b10 sp=0xc0017a9ad8 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca742ea200, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0017a9b30 sp=0xc0017a9b10 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc001b6e500?, 0xc001ea6000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0017a9b58 sp=0xc0017a9b30 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc001b6e500, {0xc001ea6000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0017a9bf0 sp=0xc0017a9b58 pc=0x1842cd9 net.(*netFD).Read(0xc001b6e500, {0xc001ea6000?, 0x1040100000000?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0017a9c38 sp=0xc0017a9bf0 pc=0x1a69a29 net.(*conn).Read(0xc000334858, {0xc001ea6000?, 0x412d040?, 0xc0017a9d68?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0017a9c80 sp=0xc0017a9c38 pc=0x1a7ea05 net.(*TCPConn).Read(0x1010401?, {0xc001ea6000?, 0x17569ca?, 0x1010401?}) :1 +0x29 fp=0xc0017a9cb0 sp=0xc0017a9c80 pc=0x1a95569 bufio.(*Reader).Read(0xc001b6ce40, {0xc0005f9c40, 0x9, 0x30?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0017a9ce8 sp=0xc0017a9cb0 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc001b6ce40}, {0xc0005f9c40, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0017a9d30 sp=0xc0017a9ce8 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc0005f9c40?, 0x9?, 0xc001e2c270?}, {0x4bd2e40?, 0xc001b6ce40?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc0017a9d80 sp=0xc0017a9d30 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005f9c00) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc0017a9e30 sp=0xc0017a9d80 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001b60b40, 0xc001b5c2f0?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x257 fp=0xc0017a9fc0 sp=0xc0017a9e30 pc=0x1e19d37 google.golang.org/grpc/internal/transport.newHTTP2Client.func11() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2a fp=0xc0017a9fe0 sp=0xc0017a9fc0 pc=0x1e0f5ea runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0017a9fe8 sp=0xc0017a9fe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1e2a

goroutine 479 [select]: runtime.gopark(0xc0012a37b0?, 0x2?, 0x98?, 0xad?, 0xc0012a3794?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012a3618 sp=0xc0012a35f8 pc=0x17870d6 runtime.selectgo(0xc0012a37b0, 0xc0012a3790, 0x0?, 0x0, 0x1f61cc0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012a3758 sp=0xc0012a3618 pc=0x1797bfe google.golang.org/grpc.newClientStreamWithParams.func4() /go/pkg/mod/google.golang.org/[email protected]/stream.go:375 +0x92 fp=0xc0012a37e0 sp=0xc0012a3758 pc=0x1eaea92 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012a37e8 sp=0xc0012a37e0 pc=0x17bd9a1 created by google.golang.org/grpc.newClientStreamWithParams /go/pkg/mod/google.golang.org/[email protected]/stream.go:374 +0xf2a

goroutine 536 [select]: runtime.gopark(0xc000d73f90?, 0x2?, 0x18?, 0x3e?, 0xc000d73f3c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000d73db8 sp=0xc000d73d98 pc=0x17870d6 runtime.selectgo(0xc000d73f90, 0xc000d73f38, 0x0?, 0x1, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000d73ef8 sp=0xc000d73db8 pc=0x1797bfe github.com/milvus-io/milvus/internal/querynodev2/tasks.(*scheduler).schedule(0xc0016705a0) /go/src/github.com/milvus-io/milvus/internal/querynodev2/tasks/concurrent_safe_scheduler.go:117 +0x159 fp=0xc000d73fc8 sp=0xc000d73ef8 pc=0x3c640d9 github.com/milvus-io/milvus/internal/querynodev2/tasks.(*scheduler).Start.func2() /go/src/github.com/milvus-io/milvus/internal/querynodev2/tasks/concurrent_safe_scheduler.go:91 +0x26 fp=0xc000d73fe0 sp=0xc000d73fc8 pc=0x3c63e66 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000d73fe8 sp=0xc000d73fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/querynodev2/tasks.(*scheduler).Start /go/src/github.com/milvus-io/milvus/internal/querynodev2/tasks/concurrent_safe_scheduler.go:91 +0xaa

goroutine 538 [select]: runtime.gopark(0xc0017afea0?, 0x2?, 0xca?, 0x69?, 0xc0017afe5c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0017afcc0 sp=0xc0017afca0 pc=0x17870d6 runtime.selectgo(0xc0017afea0, 0xc0017afe58, 0x3?, 0x0, 0x174f010?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0017afe00 sp=0xc0017afcc0 pc=0x1797bfe github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).bgCheckChannelsWork(0xc001222c30, {0x4bfad98, 0xc00054c8c0}) /go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:274 +0x126 fp=0xc0017aff98 sp=0xc0017afe00 pc=0x34d66e6 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).bgCheckChannelsWork-fm({0x4bfad98?, 0xc00054c8c0?}) :1 +0x39 fp=0xc0017affc0 sp=0xc0017aff98 pc=0x356a919 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup.func2() /go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:175 +0x2e fp=0xc0017affe0 sp=0xc0017affc0 pc=0x34d59ee runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0017affe8 sp=0xc0017affe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup /go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:175 +0x91e

goroutine 710 [select]: runtime.gopark(0xc000d76f98?, 0x2?, 0xc8?, 0x40?, 0xc000d76f7c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000d76e00 sp=0xc000d76de0 pc=0x17870d6 runtime.selectgo(0xc000d76f98, 0xc000d76f78, 0xc001a39260?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000d76f40 sp=0xc000d76e00 pc=0x1797bfe google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0018fa4c0) /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:115 +0x73 fp=0xc000d76fc8 sp=0xc000d76f40 pc=0x1e8c193 google.golang.org/grpc.newCCBalancerWrapper.func1() /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x26 fp=0xc000d76fe0 sp=0xc000d76fc8 pc=0x1e8c0e6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000d76fe8 sp=0xc000d76fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:76 +0x22a

goroutine 730 [select]: runtime.gopark(0xc001b4bf08?, 0x2?, 0xa3?, 0x50?, 0xc001b4bedc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001b4bd68 sp=0xc001b4bd48 pc=0x17870d6 runtime.selectgo(0xc001b4bf08, 0xc001b4bed8, 0x1a95869?, 0x0, 0xc001eae000?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001b4bea8 sp=0xc001b4bd68 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001b71220, 0x1) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x115 fp=0xc001b4bf38 sp=0xc001b4bea8 pc=0x1e04b95 google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000fd82a0) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x91 fp=0xc001b4bf98 sp=0xc001b4bf38 pc=0x1e052f1 google.golang.org/grpc/internal/transport.newHTTP2Client.func6() /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x7b fp=0xc001b4bfe0 sp=0xc001b4bf98 pc=0x1e0f4bb runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001b4bfe8 sp=0xc001b4bfe0 pc=0x17bd9a1 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x23fc

goroutine 509 [select]: runtime.gopark(0xc00123d948?, 0x2?, 0xa7?, 0x6d?, 0xc00123d914?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00123d798 sp=0xc00123d778 pc=0x17870d6 runtime.selectgo(0xc00123d948, 0xc00123d910, 0x18aa0d1?, 0x0, 0xc001149ad0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00123d8d8 sp=0xc00123d798 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000657bd0, {0xc0006625b0, 0x5, 0x5}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:195 +0xa5 fp=0xc00123d978 sp=0xc00123d8d8 pc=0x1e29bc5 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000657bd0, {0xc0006625b0?, 0xc00067a5d0?, 0xc00123d9c0?}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:175 +0x147 fp=0xc00123d9b8 sp=0xc00123d978 pc=0x1e29907 google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000344d80, {0xc0006625b0?, 0xc00123da38?, 0x1e10107?}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:514 +0x32 fp=0xc00123da00 sp=0xc00123d9b8 pc=0x1e2afd2 io.ReadAtLeast({0x4bd8640, 0xc000344d80}, {0xc0006625b0, 0x5, 0x5}, 0x5) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc00123da48 sp=0xc00123da00 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000d40a20, {0xc0006625b0, 0x5, 0x5}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:498 +0xac fp=0xc00123da88 sp=0xc00123da48 pc=0x1e2af4c google.golang.org/grpc.(*parser).recvMsg(0xc0006625a0, 0x7fffffff) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:597 +0x47 fp=0xc00123db08 sp=0xc00123da88 pc=0x1e9d6c7 google.golang.org/grpc.recvAndDecompress(0xc00123dc08?, 0xc000d40a20, {0x0, 0x0}, 0x7fffffff, 0x0, {0x0, 0x0}) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:729 +0x66 fp=0xc00123dba0 sp=0xc00123db08 pc=0x1e9df26 google.golang.org/grpc.recv(0x0?, {0x7fca735b3378, 0x6757a08}, 0xc000f5b180?, {0x0?, 0x0?}, {0x4542040, 0xc00110d600}, 0x0?, 0x0, ...) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:795 +0x6e fp=0xc00123dc18 sp=0xc00123dba0 pc=0x1e9e60e google.golang.org/grpc.(*csAttempt).recvMsg(0xc000664b60, {0x4542040?, 0xc00110d600}, 0x174e9e0?) /go/pkg/mod/google.golang.org/[email protected]/stream.go:1072 +0x2cc fp=0xc00123dd48 sp=0xc00123dc18 pc=0x1eb2d8c google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x174faa5?) /go/pkg/mod/google.golang.org/[email protected]/stream.go:909 +0x25 fp=0xc00123dd78 sp=0xc00123dd48 pc=0x1eb1f45 google.golang.org/grpc.(*clientStream).withRetry(0xc000d407e0, 0xc00123de90, 0xc00123de60) /go/pkg/mod/google.golang.org/[email protected]/stream.go:744 +0x3be fp=0xc00123dde8 sp=0xc00123dd78 pc=0x1eb0b3e google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d407e0, {0x4542040?, 0xc00110d600?}) /go/pkg/mod/google.golang.org/[email protected]/stream.go:908 +0x131 fp=0xc00123dec0 sp=0xc00123dde8 pc=0x1eb1bb1 go.etcd.io/etcd/api/v3/etcdserverpb.(*leaseLeaseKeepAliveClient).Recv(0xc000f33720) /go/pkg/mod/go.etcd.io/etcd/api/[email protected]/etcdserverpb/rpc.pb.go:6853 +0x4c fp=0xc00123def0 sp=0xc00123dec0 pc=0x1ee85ec go.etcd.io/etcd/client/v3.(*lessor).recvKeepAliveLoop(0xc00047c5a0) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:456 +0x286 fp=0xc00123dfc8 sp=0xc00123def0 pc=0x1f61b46 go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func1.1() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:299 +0x26 fp=0xc00123dfe0 sp=0xc00123dfc8 pc=0x1f60766 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00123dfe8 sp=0xc00123dfe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func1 /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:299 +0x65

goroutine 510 [select]: runtime.gopark(0xc0012a8738?, 0x2?, 0xb8?, 0x36?, 0xc0012a871c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012a8580 sp=0xc0012a8560 pc=0x17870d6 runtime.selectgo(0xc0012a8738, 0xc0012a8718, 0x670cd80?, 0x0, 0x53128a738?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012a86c0 sp=0xc0012a8580 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*lessor).deadlineLoop(0xc00047c5a0) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:548 +0x85 fp=0xc0012a87c8 sp=0xc0012a86c0 pc=0x1f629c5 go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func1.2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:300 +0x26 fp=0xc0012a87e0 sp=0xc0012a87c8 pc=0x1f60706 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012a87e8 sp=0xc0012a87e0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func1 /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:300 +0xa5

goroutine 511 [select]: runtime.gopark(0xc0012b3c70?, 0x2?, 0x0?, 0x0?, 0xc0012b3b84?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012b3a10 sp=0xc0012b39f0 pc=0x17870d6 runtime.selectgo(0xc0012b3c70, 0xc0012b3b80, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012b3b50 sp=0xc0012b3a10 pc=0x1797bfe github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).processKeepAliveResponse.func1() /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:469 +0x11d fp=0xc0012b3fe0 sp=0xc0012b3b50 pc=0x341df9d runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012b3fe8 sp=0xc0012b3fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).processKeepAliveResponse /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:466 +0x8d

goroutine 481 [select]: runtime.gopark(0xc0012a57b0?, 0x2?, 0xca?, 0x69?, 0xc0012a5784?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012a5600 sp=0xc0012a55e0 pc=0x17870d6 runtime.selectgo(0xc0012a57b0, 0xc0012a5780, 0x0?, 0x0, 0xc000d4abd0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012a5740 sp=0xc0012a5600 pc=0x1797bfe context.propagateCancel.func1() /usr/local/go/src/context/context.go:333 +0xa5 fp=0xc0012a57e0 sp=0xc0012a5740 pc=0x1870f65 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012a57e8 sp=0xc0012a57e0 pc=0x17bd9a1 created by context.propagateCancel /usr/local/go/src/context/context.go:332 +0x1bc

goroutine 546 [select]: runtime.gopark(0xc001433f10?, 0x5?, 0x2?, 0x0?, 0xc0014338ae?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0014336e8 sp=0xc0014336c8 pc=0x17870d6 runtime.selectgo(0xc001433f10, 0xc0014338a4, 0xc0014422c0?, 0x0, 0x3faa6e0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001433828 sp=0xc0014336e8 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*watchGrpcStream).run(0xc000664c30) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:537 +0x354 fp=0xc001433fc8 sp=0xc001433828 pc=0x1f6f1b4 go.etcd.io/etcd/client/v3.(*watcher).newWatcherGrpcStream.func1() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:285 +0x26 fp=0xc001433fe0 sp=0xc001433fc8 pc=0x1f6daa6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001433fe8 sp=0xc001433fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watcher).newWatcherGrpcStream /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:285 +0x34d

goroutine 537 [select]: runtime.gopark(0xc00128fa70?, 0x3?, 0x50?, 0x79?, 0xc00128f9b2?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00128f800 sp=0xc00128f7e0 pc=0x17870d6 runtime.selectgo(0xc00128fa70, 0xc00128f9ac, 0xc?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00128f940 sp=0xc00128f800 pc=0x1797bfe github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop(0xc001222c30, {0x4bfad98?, 0xc00054c8c0}, 0xffffffffffffffff) /go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:696 +0x297 fp=0xc00128ff88 sp=0xc00128f940 pc=0x34dbbd7 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop-fm({0x4bfad98?, 0xc00054c8c0?}, 0xc0012a5f98?) :1 +0x3c fp=0xc00128ffb8 sp=0xc00128ff88 pc=0x356a99c github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup.func1() /go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:170 +0x35 fp=0xc00128ffe0 sp=0xc00128ffb8 pc=0x34d5a55 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00128ffe8 sp=0xc00128ffe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup /go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:170 +0x85e

goroutine 548 [select]: runtime.gopark(0xc0016767b0?, 0x2?, 0x0?, 0x0?, 0xc001676794?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001676618 sp=0xc0016765f8 pc=0x17870d6 runtime.selectgo(0xc0016767b0, 0xc001676790, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001676758 sp=0xc001676618 pc=0x1797bfe google.golang.org/grpc.newClientStreamWithParams.func4() /go/pkg/mod/google.golang.org/[email protected]/stream.go:375 +0x92 fp=0xc0016767e0 sp=0xc001676758 pc=0x1eaea92 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0016767e8 sp=0xc0016767e0 pc=0x17bd9a1 created by google.golang.org/grpc.newClientStreamWithParams /go/pkg/mod/google.golang.org/[email protected]/stream.go:374 +0xf2a

goroutine 549 [select]: runtime.gopark(0xc000dad968?, 0x2?, 0xe5?, 0x36?, 0xc000dad934?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000dad7b8 sp=0xc000dad798 pc=0x17870d6 runtime.selectgo(0xc000dad968, 0xc000dad930, 0x18aa0d1?, 0x0, 0xc000cd5eb0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000dad8f8 sp=0xc000dad7b8 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000657f40, {0xc000662700, 0x5, 0x5}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:195 +0xa5 fp=0xc000dad998 sp=0xc000dad8f8 pc=0x1e29bc5 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000657f40, {0xc000662700?, 0xc00067a600?, 0xc000dad9e0?}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:175 +0x147 fp=0xc000dad9d8 sp=0xc000dad998 pc=0x1e29907 google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0003453e0, {0xc000662700?, 0xc000dada58?, 0x1e10107?}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:514 +0x32 fp=0xc000dada20 sp=0xc000dad9d8 pc=0x1e2afd2 io.ReadAtLeast({0x4bd8640, 0xc0003453e0}, {0xc000662700, 0x5, 0x5}, 0x5) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000dada68 sp=0xc000dada20 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000d40ea0, {0xc000662700, 0x5, 0x5}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:498 +0xac fp=0xc000dadaa8 sp=0xc000dada68 pc=0x1e2af4c google.golang.org/grpc.(*parser).recvMsg(0xc0006626f0, 0x7fffffff) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:597 +0x47 fp=0xc000dadb28 sp=0xc000dadaa8 pc=0x1e9d6c7 google.golang.org/grpc.recvAndDecompress(0xc000d72c28?, 0xc000d40ea0, {0x0, 0x0}, 0x7fffffff, 0x0, {0x0, 0x0}) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:729 +0x66 fp=0xc000dadbc0 sp=0xc000dadb28 pc=0x1e9df26 google.golang.org/grpc.recv(0x0?, {0x7fca735b3378, 0x6757a08}, 0x0?, {0x0?, 0x0?}, {0x458dde0, 0xc0010759d0}, 0x0?, 0x0, ...) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:795 +0x6e fp=0xc000dadc38 sp=0xc000dadbc0 pc=0x1e9e60e google.golang.org/grpc.(*csAttempt).recvMsg(0xc000664dd0, {0x458dde0?, 0xc0010759d0}, 0xc00149d6c0?) /go/pkg/mod/google.golang.org/[email protected]/stream.go:1072 +0x2cc fp=0xc000dadd68 sp=0xc000dadc38 pc=0x1eb2d8c google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x174eb11?) /go/pkg/mod/google.golang.org/[email protected]/stream.go:909 +0x25 fp=0xc000dadd98 sp=0xc000dadd68 pc=0x1eb1f45 google.golang.org/grpc.(*clientStream).withRetry(0xc000d40c60, 0xc000dadeb0, 0xc000dade80) /go/pkg/mod/google.golang.org/[email protected]/stream.go:744 +0x3be fp=0xc000dade08 sp=0xc000dadd98 pc=0x1eb0b3e google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d40c60, {0x458dde0?, 0xc0010759d0?}) /go/pkg/mod/google.golang.org/[email protected]/stream.go:908 +0x131 fp=0xc000dadee0 sp=0xc000dade08 pc=0x1eb1bb1 go.etcd.io/etcd/api/v3/etcdserverpb.(*watchWatchClient).Recv(0xc000f33910) /go/pkg/mod/go.etcd.io/etcd/api/[email protected]/etcdserverpb/rpc.pb.go:6714 +0x4c fp=0xc000dadf10 sp=0xc000dadee0 pc=0x1ee7f0c go.etcd.io/etcd/client/v3.(*watchGrpcStream).serveWatchClient(0xc000664c30, {0x4c0e468, 0xc000f33910}) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:757 +0x6a fp=0xc000dadfb8 sp=0xc000dadf10 pc=0x1f717ca go.etcd.io/etcd/client/v3.(*watchGrpcStream).newWatchClient.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:912 +0x2e fp=0xc000dadfe0 sp=0xc000dadfb8 pc=0x1f7266e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000dadfe8 sp=0xc000dadfe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watchGrpcStream).newWatchClient /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:912 +0x5eb

goroutine 550 [select]: runtime.gopark(0xc0017a8e40?, 0x5?, 0x88?, 0x8c?, 0xc0017a8de6?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0017a8c40 sp=0xc0017a8c20 pc=0x17870d6 runtime.selectgo(0xc0017a8e40, 0xc0017a8ddc, 0x0?, 0x1, 0x41cb860?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0017a8d80 sp=0xc0017a8c40 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*watchGrpcStream).serveSubstream(0xc000664c30, 0xc0014422c0, 0xc00166e180) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:803 +0x265 fp=0xc0017a8fb8 sp=0xc0017a8d80 pc=0x1f71b65 go.etcd.io/etcd/client/v3.(*watchGrpcStream).run.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0x2e fp=0xc0017a8fe0 sp=0xc0017a8fb8 pc=0x1f70e0e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0017a8fe8 sp=0xc0017a8fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watchGrpcStream).run /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0xc2c

goroutine 551 [select]: runtime.gopark(0xc001465bf8?, 0x3?, 0xb8?, 0x26?, 0xc001465b6a?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0014659d0 sp=0xc0014659b0 pc=0x17870d6 runtime.selectgo(0xc001465bf8, 0xc001465b64, 0xc0002af300?, 0x0, 0xc000d72b78?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001465b10 sp=0xc0014659d0 pc=0x1797bfe github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).LivenessCheck.func1() /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:820 +0x215 fp=0xc001465fe0 sp=0xc001465b10 pc=0x3422a75 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001465fe8 sp=0xc001465fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).LivenessCheck /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:808 +0x22c

goroutine 553 [select]: runtime.gopark(0xc001677f80?, 0x2?, 0xca?, 0x69?, 0xc001677f54?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001677dc0 sp=0xc001677da0 pc=0x17870d6 runtime.selectgo(0xc001677f80, 0xc001677f50, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001677f00 sp=0xc001677dc0 pc=0x1797bfe github.com/panjf2000/ants/v2.(*Pool).purgeStaleWorkers(0xc001498be0, {0x4bfad98, 0xc000592730}) /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:83 +0x10a fp=0xc001677fb8 sp=0xc001677f00 pc=0x2c0246a github.com/panjf2000/ants/v2.(*Pool).goPurge.func1() /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:147 +0x2e fp=0xc001677fe0 sp=0xc001677fb8 pc=0x2c02a6e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001677fe8 sp=0xc001677fe0 pc=0x17bd9a1 created by github.com/panjf2000/ants/v2.(*Pool).goPurge /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:147 +0xe5

goroutine 554 [select]: runtime.gopark(0xc001678780?, 0x2?, 0xca?, 0x69?, 0xc001678744?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0016785c8 sp=0xc0016785a8 pc=0x17870d6 runtime.selectgo(0xc001678780, 0xc001678740, 0xc0010f5218?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001678708 sp=0xc0016785c8 pc=0x1797bfe github.com/panjf2000/ants/v2.(*Pool).ticktock(0xc001498be0, {0x4bfad98, 0xc000592780}) /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:125 +0x145 fp=0xc0016787b8 sp=0xc001678708 pc=0x2c02805 github.com/panjf2000/ants/v2.(*Pool).goTicktock.func1() /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:154 +0x2e fp=0xc0016787e0 sp=0xc0016787b8 pc=0x2c02c0e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0016787e8 sp=0xc0016787e0 pc=0x17bd9a1 created by github.com/panjf2000/ants/v2.(*Pool).goTicktock /go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:154 +0x115

goroutine 640 [IO wait]: runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0x22?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000467aa0 sp=0xc000467a80 pc=0x17870d6 runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000467ad8 sp=0xc000467aa0 pc=0x177f2f7 internal/poll.runtime_pollWait(0x7fca742ea2f0, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000467af8 sp=0xc000467ad8 pc=0x17b67a9 internal/poll.(*pollDesc).wait(0xc001140100?, 0xc00198a000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000467b20 sp=0xc000467af8 pc=0x1841772 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc001140100, {0xc00198a000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000467bb8 sp=0xc000467b20 pc=0x1842cd9 net.(*netFD).Read(0xc001140100, {0xc00198a000?, 0x1040100000000?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000467c00 sp=0xc000467bb8 pc=0x1a69a29 net.(*conn).Read(0xc001428540, {0xc00198a000?, 0x0?, 0x0?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000467c48 sp=0xc000467c00 pc=0x1a7ea05 net.(*TCPConn).Read(0x1010401?, {0xc00198a000?, 0x17569ca?, 0x1010401?}) :1 +0x29 fp=0xc000467c78 sp=0xc000467c48 pc=0x1a95569 bufio.(*Reader).Read(0xc00054b0e0, {0xc00046f380, 0x9, 0x30?}) /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000467cb0 sp=0xc000467c78 pc=0x18f7e1b io.ReadAtLeast({0x4bd2e40, 0xc00054b0e0}, {0xc00046f380, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000467cf8 sp=0xc000467cb0 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 golang.org/x/net/http2.readFrameHeader({0xc00046f380?, 0x9?, 0xc001f04690?}, {0x4bd2e40?, 0xc00054b0e0?}) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc000467d48 sp=0xc000467cf8 pc=0x1ddd14e golang.org/x/net/http2.(*Framer).ReadFrame(0xc00046f340) /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc000467df8 sp=0xc000467d48 pc=0x1dddef5 google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00057c9c0, 0x0?, 0xc0000fbaa0?) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:637 +0x174 fp=0xc000467f08 sp=0xc000467df8 pc=0x1e20674 google.golang.org/grpc.(*Server).serveStreams(0xc000cc61e0, {0x4c10380?, 0xc00057c9c0}) /go/pkg/mod/google.golang.org/[email protected]/server.go:950 +0x189 fp=0xc000467f80 sp=0xc000467f08 pc=0x1ea2529 google.golang.org/grpc.(*Server).handleRawConn.func1() /go/pkg/mod/google.golang.org/[email protected]/server.go:892 +0x46 fp=0xc000467fe0 sp=0xc000467f80 pc=0x1ea1da6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000467fe8 sp=0xc000467fe0 pc=0x17bd9a1 created by google.golang.org/grpc.(*Server).handleRawConn /go/pkg/mod/google.golang.org/[email protected]/server.go:891 +0x185

goroutine 561 [select]: runtime.gopark(0xc001673760?, 0x3?, 0xca?, 0x69?, 0xc00167371a?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001673568 sp=0xc001673548 pc=0x17870d6 runtime.selectgo(0xc001673760, 0xc001673714, 0x7?, 0x0, 0xc001673738?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0016736a8 sp=0xc001673568 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*lessor).keepAliveCtxCloser(0xc0014500a0, {0x4bfad98, 0xc000592fa0}, 0xc000f571c0?, 0xc00166e840) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:330 +0xd9 fp=0xc0016737a8 sp=0xc0016736a8 pc=0x1f60ab9 go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:297 +0x36 fp=0xc0016737e0 sp=0xc0016737a8 pc=0x1f607d6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0016737e8 sp=0xc0016737e0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*lessor).KeepAlive /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:297 +0x5aa

goroutine 496 [select]: runtime.gopark(0xc0012a5e40?, 0x5?, 0x88?, 0x5c?, 0xc0012a5de6?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012a5c40 sp=0xc0012a5c20 pc=0x17870d6 runtime.selectgo(0xc0012a5e40, 0xc0012a5ddc, 0x7fca735186b8?, 0x1, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012a5d80 sp=0xc0012a5c40 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*watchGrpcStream).serveSubstream(0xc0013e9790, 0xc0004c2420, 0xc0017f6e40) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:803 +0x265 fp=0xc0012a5fb8 sp=0xc0012a5d80 pc=0x1f71b65 go.etcd.io/etcd/client/v3.(*watchGrpcStream).run.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0x2e fp=0xc0012a5fe0 sp=0xc0012a5fb8 pc=0x1f70e0e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012a5fe8 sp=0xc0012a5fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watchGrpcStream).run /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0xc2c

goroutine 535 [chan receive]: runtime.gopark(0xd3?, 0x53eb8b9?, 0x48?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00066fd68 sp=0xc00066fd48 pc=0x17870d6 runtime.chanrecv(0xc00166e540, 0xc00066fea0, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00066fdf8 sp=0xc00066fd68 pc=0x174f5bd runtime.chanrecv2(0x46d4457?, 0x0?) /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00066fe20 sp=0xc00066fdf8 pc=0x174f0f8 github.com/milvus-io/milvus/internal/querynodev2/tasks.(*scheduler).exec(0xc0016705a0) /go/src/github.com/milvus-io/milvus/internal/querynodev2/tasks/concurrent_safe_scheduler.go:213 +0xae fp=0xc00066ffc8 sp=0xc00066fe20 pc=0x3c6480e github.com/milvus-io/milvus/internal/querynodev2/tasks.(*scheduler).Start.func1() /go/src/github.com/milvus-io/milvus/internal/querynodev2/tasks/concurrent_safe_scheduler.go:88 +0x26 fp=0xc00066ffe0 sp=0xc00066ffc8 pc=0x3c63ec6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00066ffe8 sp=0xc00066ffe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/querynodev2/tasks.(*scheduler).Start /go/src/github.com/milvus-io/milvus/internal/querynodev2/tasks/concurrent_safe_scheduler.go:88 +0x6d

goroutine 378 [select]: runtime.gopark(0xc001679728?, 0x2?, 0xa0?, 0x95?, 0xc00167971c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001679548 sp=0xc001679528 pc=0x17870d6 runtime.selectgo(0xc001679728, 0xc001679718, 0x17?, 0x0, 0x46e5f86?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001679688 sp=0xc001679548 pc=0x1797bfe github.com/milvus-io/milvus/internal/util/sessionutil.(*sessionWatcher).start.func1() /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:657 +0x129 fp=0xc0016797e0 sp=0xc001679688 pc=0x3420bc9 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0016797e8 sp=0xc0016797e0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/util/sessionutil.(*sessionWatcher).start /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:655 +0x56

goroutine 562 [select]: runtime.gopark(0xc000693948?, 0x2?, 0xa7?, 0x6d?, 0xc000693914?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000693798 sp=0xc000693778 pc=0x17870d6 runtime.selectgo(0xc000693948, 0xc000693910, 0x18aa0d1?, 0x0, 0xc001149ae0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0006938d8 sp=0xc000693798 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000593450, {0xc000663858, 0x5, 0x5}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:195 +0xa5 fp=0xc000693978 sp=0xc0006938d8 pc=0x1e29bc5 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000593450, {0xc000663858?, 0xc00067b0b0?, 0xc0006939c0?}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:175 +0x147 fp=0xc0006939b8 sp=0xc000693978 pc=0x1e29907 google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001228cc0, {0xc000663858?, 0xc000693a38?, 0x1e10107?}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:514 +0x32 fp=0xc000693a00 sp=0xc0006939b8 pc=0x1e2afd2 io.ReadAtLeast({0x4bd8640, 0xc001228cc0}, {0xc000663858, 0x5, 0x5}, 0x5) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000693a48 sp=0xc000693a00 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000d41b00, {0xc000663858, 0x5, 0x5}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:498 +0xac fp=0xc000693a88 sp=0xc000693a48 pc=0x1e2af4c google.golang.org/grpc.(*parser).recvMsg(0xc000663848, 0x7fffffff) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:597 +0x47 fp=0xc000693b08 sp=0xc000693a88 pc=0x1e9d6c7 google.golang.org/grpc.recvAndDecompress(0xc000693c08?, 0xc000d41b00, {0x0, 0x0}, 0x7fffffff, 0x0, {0x0, 0x0}) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:729 +0x66 fp=0xc000693ba0 sp=0xc000693b08 pc=0x1e9df26 google.golang.org/grpc.recv(0x0?, {0x7fca735b3378, 0x6757a08}, 0x0?, {0x0?, 0x0?}, {0x4542040, 0xc00110d680}, 0x0?, 0x0, ...) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:795 +0x6e fp=0xc000693c18 sp=0xc000693ba0 pc=0x1e9e60e google.golang.org/grpc.(*csAttempt).recvMsg(0xc001222b60, {0x4542040?, 0xc00110d680}, 0x174e9e0?) /go/pkg/mod/google.golang.org/[email protected]/stream.go:1072 +0x2cc fp=0xc000693d48 sp=0xc000693c18 pc=0x1eb2d8c google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x174faa5?) /go/pkg/mod/google.golang.org/[email protected]/stream.go:909 +0x25 fp=0xc000693d78 sp=0xc000693d48 pc=0x1eb1f45 google.golang.org/grpc.(*clientStream).withRetry(0xc000d418c0, 0xc000693e90, 0xc000693e60) /go/pkg/mod/google.golang.org/[email protected]/stream.go:744 +0x3be fp=0xc000693de8 sp=0xc000693d78 pc=0x1eb0b3e google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d418c0, {0x4542040?, 0xc00110d680?}) /go/pkg/mod/google.golang.org/[email protected]/stream.go:908 +0x131 fp=0xc000693ec0 sp=0xc000693de8 pc=0x1eb1bb1 go.etcd.io/etcd/api/v3/etcdserverpb.(*leaseLeaseKeepAliveClient).Recv(0xc000fa1d20) /go/pkg/mod/go.etcd.io/etcd/api/[email protected]/etcdserverpb/rpc.pb.go:6853 +0x4c fp=0xc000693ef0 sp=0xc000693ec0 pc=0x1ee85ec go.etcd.io/etcd/client/v3.(*lessor).recvKeepAliveLoop(0xc0014500a0) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:456 +0x286 fp=0xc000693fc8 sp=0xc000693ef0 pc=0x1f61b46 go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func1.1() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:299 +0x26 fp=0xc000693fe0 sp=0xc000693fc8 pc=0x1f60766 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000693fe8 sp=0xc000693fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func1 /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:299 +0x65

goroutine 494 [select]: runtime.gopark(0xc0012b0f10?, 0x2?, 0x0?, 0x0?, 0xc0012b0ed4?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012b0d58 sp=0xc0012b0d38 pc=0x17870d6 runtime.selectgo(0xc0012b0f10, 0xc0012b0ed0, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012b0e98 sp=0xc0012b0d58 pc=0x1797bfe github.com/milvus-io/milvus/internal/metastore/kv/rootcoord.(*SuffixSnapshot).startBackgroundGC(0xc00035b580) /go/src/github.com/milvus-io/milvus/internal/metastore/kv/rootcoord/suffix_snapshot.go:560 +0x106 fp=0xc0012b0fc8 sp=0xc0012b0e98 pc=0x3cb9546 github.com/milvus-io/milvus/internal/metastore/kv/rootcoord.NewSuffixSnapshot.func1() /go/src/github.com/milvus-io/milvus/internal/metastore/kv/rootcoord/suffix_snapshot.go:124 +0x26 fp=0xc0012b0fe0 sp=0xc0012b0fc8 pc=0x3cb52c6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012b0fe8 sp=0xc0012b0fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/metastore/kv/rootcoord.NewSuffixSnapshot /go/src/github.com/milvus-io/milvus/internal/metastore/kv/rootcoord/suffix_snapshot.go:124 +0x34a

goroutine 563 [select]: runtime.gopark(0xc001674738?, 0x2?, 0x85?, 0xca?, 0xc00167471c?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001674580 sp=0xc001674560 pc=0x17870d6 runtime.selectgo(0xc001674738, 0xc001674718, 0x670cd80?, 0x0, 0x532f8b4f8?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0016746c0 sp=0xc001674580 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*lessor).deadlineLoop(0xc0014500a0) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:548 +0x85 fp=0xc0016747c8 sp=0xc0016746c0 pc=0x1f629c5 go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func1.2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:300 +0x26 fp=0xc0016747e0 sp=0xc0016747c8 pc=0x1f60706 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0016747e8 sp=0xc0016747e0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func1 /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:300 +0xa5

goroutine 564 [select]: runtime.gopark(0xc000462c70?, 0x2?, 0x0?, 0x0?, 0xc000462b84?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000462a10 sp=0xc0004629f0 pc=0x17870d6 runtime.selectgo(0xc000462c70, 0xc000462b80, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000462b50 sp=0xc000462a10 pc=0x1797bfe github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).processKeepAliveResponse.func1() /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:469 +0x11d fp=0xc000462fe0 sp=0xc000462b50 pc=0x341df9d runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000462fe8 sp=0xc000462fe0 pc=0x17bd9a1 created by github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).processKeepAliveResponse /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:466 +0x8d

goroutine 565 [select]: runtime.gopark(0xc001673fb0?, 0x2?, 0x98?, 0xad?, 0xc001673f94?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001673e18 sp=0xc001673df8 pc=0x17870d6 runtime.selectgo(0xc001673fb0, 0xc001673f90, 0x0?, 0x0, 0x1f61cc0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001673f58 sp=0xc001673e18 pc=0x1797bfe google.golang.org/grpc.newClientStreamWithParams.func4() /go/pkg/mod/google.golang.org/[email protected]/stream.go:375 +0x92 fp=0xc001673fe0 sp=0xc001673f58 pc=0x1eaea92 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001673fe8 sp=0xc001673fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newClientStreamWithParams /go/pkg/mod/google.golang.org/[email protected]/stream.go:374 +0xf2a

goroutine 566 [select]: runtime.gopark(0xc000463f08?, 0x4?, 0x5?, 0x0?, 0xc000463ed8?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000463d18 sp=0xc000463cf8 pc=0x17870d6 runtime.selectgo(0xc000463f08, 0xc000463ed0, 0x670cd80?, 0x0, 0x533b268b8?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000463e58 sp=0xc000463d18 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*lessor).sendKeepAliveLoop(0xc0014500a0, {0x4c0e360, 0xc000fa1d20}) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:588 +0x1e9 fp=0xc000463fb8 sp=0xc000463e58 pc=0x1f62d69 go.etcd.io/etcd/client/v3.(*lessor).resetRecv.func3() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:498 +0x2e fp=0xc000463fe0 sp=0xc000463fb8 pc=0x1f6228e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000463fe8 sp=0xc000463fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*lessor).resetRecv /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/lease.go:498 +0x2cd

goroutine 539 [select]: runtime.gopark(0xc001676fb0?, 0x2?, 0x0?, 0x0?, 0xc001676f84?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001676e00 sp=0xc001676de0 pc=0x17870d6 runtime.selectgo(0xc001676fb0, 0xc001676f80, 0x0?, 0x0, 0xc0016705c8?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001676f40 sp=0xc001676e00 pc=0x1797bfe context.propagateCancel.func1() /usr/local/go/src/context/context.go:333 +0xa5 fp=0xc001676fe0 sp=0xc001676f40 pc=0x1870f65 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001676fe8 sp=0xc001676fe0 pc=0x17bd9a1 created by context.propagateCancel /usr/local/go/src/context/context.go:332 +0x1bc

goroutine 540 [select]: runtime.gopark(0xc000da9f10?, 0x5?, 0x2?, 0x0?, 0xc000da98ae?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000da96e8 sp=0xc000da96c8 pc=0x17870d6 runtime.selectgo(0xc000da9f10, 0xc000da98a4, 0xc0004c2420?, 0x0, 0x3faa6e0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000da9828 sp=0xc000da96e8 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*watchGrpcStream).run(0xc0013e9790) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:537 +0x354 fp=0xc000da9fc8 sp=0xc000da9828 pc=0x1f6f1b4 go.etcd.io/etcd/client/v3.(*watcher).newWatcherGrpcStream.func1() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:285 +0x26 fp=0xc000da9fe0 sp=0xc000da9fc8 pc=0x1f6daa6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000da9fe8 sp=0xc000da9fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watcher).newWatcherGrpcStream /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:285 +0x34d

goroutine 495 [select]: runtime.gopark(0xc000d72e40?, 0x5?, 0x88?, 0x84?, 0xc000d72de6?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000d72c40 sp=0xc000d72c20 pc=0x17870d6 runtime.selectgo(0xc000d72e40, 0xc000d72ddc, 0x0?, 0x1, 0x41cb860?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000d72d80 sp=0xc000d72c40 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*watchGrpcStream).serveSubstream(0xc0013e9790, 0xc0004c2370, 0xc0017f6e40) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:803 +0x265 fp=0xc000d72fb8 sp=0xc000d72d80 pc=0x1f71b65 go.etcd.io/etcd/client/v3.(*watchGrpcStream).run.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0x2e fp=0xc000d72fe0 sp=0xc000d72fb8 pc=0x1f70e0e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000d72fe8 sp=0xc000d72fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watchGrpcStream).run /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0xc2c

goroutine 542 [select]: runtime.gopark(0xc0017e8fb0?, 0x2?, 0x0?, 0x0?, 0xc0017e8f94?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0017e8e18 sp=0xc0017e8df8 pc=0x17870d6 runtime.selectgo(0xc0017e8fb0, 0xc0017e8f90, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0017e8f58 sp=0xc0017e8e18 pc=0x1797bfe google.golang.org/grpc.newClientStreamWithParams.func4() /go/pkg/mod/google.golang.org/[email protected]/stream.go:375 +0x92 fp=0xc0017e8fe0 sp=0xc0017e8f58 pc=0x1eaea92 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0017e8fe8 sp=0xc0017e8fe0 pc=0x17bd9a1 created by google.golang.org/grpc.newClientStreamWithParams /go/pkg/mod/google.golang.org/[email protected]/stream.go:374 +0xf2a

goroutine 543 [runnable]: runtime.gopark(0xc001773968?, 0x2?, 0xe5?, 0x36?, 0xc001773934?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0017737b8 sp=0xc001773798 pc=0x17870d6 runtime.selectgo(0xc001773968, 0xc001773930, 0x18aa0d1?, 0x0, 0xc001ffc870?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0017738f8 sp=0xc0017737b8 pc=0x1797bfe google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00054ca50, {0xc0015eec10, 0x5, 0x5}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:195 +0xa5 fp=0xc001773998 sp=0xc0017738f8 pc=0x1e29bc5 google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00054ca50, {0xc0015eec10?, 0xc0015f47f8?, 0xc0017739e0?}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:175 +0x147 fp=0xc0017739d8 sp=0xc001773998 pc=0x1e29907 google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000cd6780, {0xc0015eec10?, 0xc001773a58?, 0x1e10107?}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:514 +0x32 fp=0xc001773a20 sp=0xc0017739d8 pc=0x1e2afd2 io.ReadAtLeast({0x4bd8640, 0xc000cd6780}, {0xc0015eec10, 0x5, 0x5}, 0x5) /usr/local/go/src/io/io.go:332 +0x9a fp=0xc001773a68 sp=0xc001773a20 pc=0x18158fa io.ReadFull(...) /usr/local/go/src/io/io.go:351 google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0017b38c0, {0xc0015eec10, 0x5, 0x5}) /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:498 +0xac fp=0xc001773aa8 sp=0xc001773a68 pc=0x1e2af4c google.golang.org/grpc.(*parser).recvMsg(0xc0015eec00, 0x7fffffff) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:597 +0x47 fp=0xc001773b28 sp=0xc001773aa8 pc=0x1e9d6c7 google.golang.org/grpc.recvAndDecompress(0xc001773c28?, 0xc0017b38c0, {0x0, 0x0}, 0x7fffffff, 0x0, {0x0, 0x0}) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:729 +0x66 fp=0xc001773bc0 sp=0xc001773b28 pc=0x1e9df26 google.golang.org/grpc.recv(0x0?, {0x7fca735b3378, 0x6757a08}, 0x0?, {0x0?, 0x0?}, {0x458dde0, 0xc001d21180}, 0x0?, 0x0, ...) /go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:795 +0x6e fp=0xc001773c38 sp=0xc001773bc0 pc=0x1e9e60e google.golang.org/grpc.(*csAttempt).recvMsg(0xc0013e9930, {0x458dde0?, 0xc001d21180}, 0xc000cda340?) /go/pkg/mod/google.golang.org/[email protected]/stream.go:1072 +0x2cc fp=0xc001773d68 sp=0xc001773c38 pc=0x1eb2d8c google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x174eb11?) /go/pkg/mod/google.golang.org/[email protected]/stream.go:909 +0x25 fp=0xc001773d98 sp=0xc001773d68 pc=0x1eb1f45 google.golang.org/grpc.(*clientStream).withRetry(0xc0017b3680, 0xc001773eb0, 0xc001773e80) /go/pkg/mod/google.golang.org/[email protected]/stream.go:744 +0x3be fp=0xc001773e08 sp=0xc001773d98 pc=0x1eb0b3e google.golang.org/grpc.(*clientStream).RecvMsg(0xc0017b3680, {0x458dde0?, 0xc001d21180?}) /go/pkg/mod/google.golang.org/[email protected]/stream.go:908 +0x131 fp=0xc001773ee0 sp=0xc001773e08 pc=0x1eb1bb1 go.etcd.io/etcd/api/v3/etcdserverpb.(*watchWatchClient).Recv(0xc0013edb70) /go/pkg/mod/go.etcd.io/etcd/api/[email protected]/etcdserverpb/rpc.pb.go:6714 +0x4c fp=0xc001773f10 sp=0xc001773ee0 pc=0x1ee7f0c go.etcd.io/etcd/client/v3.(*watchGrpcStream).serveWatchClient(0xc0013e9790, {0x4c0e468, 0xc0013edb70}) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:757 +0x6a fp=0xc001773fb8 sp=0xc001773f10 pc=0x1f717ca go.etcd.io/etcd/client/v3.(*watchGrpcStream).newWatchClient.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:912 +0x2e fp=0xc001773fe0 sp=0xc001773fb8 pc=0x1f7266e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001773fe8 sp=0xc001773fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watchGrpcStream).newWatchClient /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:912 +0x5eb

goroutine 544 [select]: runtime.gopark(0xc0012ace40?, 0x5?, 0x88?, 0xcc?, 0xc0012acde6?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0012acc40 sp=0xc0012acc20 pc=0x17870d6 runtime.selectgo(0xc0012ace40, 0xc0012acddc, 0x0?, 0x1, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0012acd80 sp=0xc0012acc40 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*watchGrpcStream).serveSubstream(0xc0013e9790, 0xc001a80000, 0xc0017f6e40) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:803 +0x265 fp=0xc0012acfb8 sp=0xc0012acd80 pc=0x1f71b65 go.etcd.io/etcd/client/v3.(*watchGrpcStream).run.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0x2e fp=0xc0012acfe0 sp=0xc0012acfb8 pc=0x1f70e0e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0012acfe8 sp=0xc0012acfe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watchGrpcStream).run /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0xc2c

goroutine 545 [select]: runtime.gopark(0xc001777e40?, 0x5?, 0x88?, 0x7c?, 0xc001777de6?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001777c40 sp=0xc001777c20 pc=0x17870d6 runtime.selectgo(0xc001777e40, 0xc001777ddc, 0x1?, 0x1, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001777d80 sp=0xc001777c40 pc=0x1797bfe go.etcd.io/etcd/client/v3.(*watchGrpcStream).serveSubstream(0xc0013e9790, 0xc001a800b0, 0xc0017f6e40) /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:803 +0x265 fp=0xc001777fb8 sp=0xc001777d80 pc=0x1f71b65 go.etcd.io/etcd/client/v3.(*watchGrpcStream).run.func2() /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0x2e fp=0xc001777fe0 sp=0xc001777fb8 pc=0x1f70e0e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001777fe8 sp=0xc001777fe0 pc=0x17bd9a1 created by go.etcd.io/etcd/client/v3.(*watchGrpcStream).run /go/pkg/mod/go.etcd.io/etcd/client/[email protected]/watch.go:554 +0xc2c

Dec 26 '23 08:12 zzggfly

/assign @congqixia /unassign

Dec 27 '23 01:12 yanliang567

同样的问题，请问您解决了吗

Jan 31 '24 05:01 whNLP

please try 2.3.7 see if it fixed

Jan 31 '24 05:01 xiaofan-luan

@xiaofan-luan I had the same issue, It was working fine for 5 days and today just started getting this error, I re created new docker container using the latest image 2.3.7 and it is working fine.

Feb 05 '24 11:02 arrrrny

What docker command did you use to run the image?

Feb 09 '24 18:02 nutmilk10

@nutmilk10

sudo docker run -d \                           
--name milvus-standalone\
--security-opt seccomp:unconfined \
-e ETCD_USE_EMBED=true \
-e ETCD_DATA_DIR=/var/lib/milvus/etcd \
-e ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml \
-e COMMON_STORAGETYPE=local \
-v ${DOCKER_VOLUME_DIRECTORY:-}./milvus:/var/lib/milvus \
-v ${DOCKER_VOLUME_DIRECTORY:-}./embedEtcd.yaml:/milvus/configs/embedEtcd.yaml \
-p 19530:19530 \
-p 9091:9091 \
--health-cmd="curl -f http://localhost:9091/healthz" \
--health-interval=30s \
--health-start-period=90s \
--health-timeout=20s \
--health-retries=3 \
milvusdb/milvus \
milvus run standalone

The bottom milvusdb/milvus pulls the latest image from docker which is 2.3.7

Feb 09 '24 18:02 arrrrny

@nutmilk10 I have the same laptop, 2.3.8 would be fine. Make sure to delete the old container built. When you build a new docker container by pulling the latest milvus it just works fine. Recommend restarting docker and resetting kubernetes inside docker and they are all green before building the docker container

Feb 12 '24 17:02 arrrrny

I'm getting this issue in Docker,

...

2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x58 fp=0x40017bcd60 sp=0x40017bcd10 pc=0x211f1b8
2024-02-15 14:06:00 milvus-standalone  | golang.org/x/net/http2.(*Framer).ReadFrame(0x40011aa620)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x80 fp=0x40017bce10 sp=0x40017bcd60 pc=0x211fe50
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*http2Client).reader(0x4001658b40, 0x0?)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x1b4 fp=0x40017bcfb0 sp=0x40017bce10 pc=0x2157b94
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.newHTTP2Client.func11()
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2c fp=0x40017bcfd0 sp=0x40017bcfb0 pc=0x214e4dc
2024-02-15 14:06:00 milvus-standalone  | runtime.goexit()
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40017bcfd0 sp=0x40017bcfd0 pc=0x1b9a5c4
2024-02-15 14:06:00 milvus-standalone  | created by google.golang.org/grpc/internal/transport.newHTTP2Client
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1638
2024-02-15 14:06:00 milvus-standalone  | 
2024-02-15 14:06:00 milvus-standalone  | goroutine 418 [select]:
2024-02-15 14:06:00 milvus-standalone  | runtime.gopark(0x4000ccdef8?, 0x2?, 0x9?, 0x0?, 0x4000ccdec4?)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0x4000ccdd70 sp=0x4000ccdd50 pc=0x1b651d0
2024-02-15 14:06:00 milvus-standalone  | runtime.selectgo(0x4000ccdef8, 0x4000ccdec0, 0x4000ccdec8?, 0x0, 0x400059cef0?, 0x1)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/select.go:327 +0x68c fp=0x4000ccde80 sp=0x4000ccdd70 pc=0x1b7608c
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x400163b3b0, 0x1)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x14c fp=0x4000ccdf20 sp=0x4000ccde80 pc=0x2144acc
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x4000fe0230)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x7c fp=0x4000ccdf80 sp=0x4000ccdf20 pc=0x214532c
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x68 fp=0x4000ccdfd0 sp=0x4000ccdf80 pc=0x214e3e8
2024-02-15 14:06:00 milvus-standalone  | runtime.goexit()
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000ccdfd0 sp=0x4000ccdfd0 pc=0x1b9a5c4
2024-02-15 14:06:00 milvus-standalone  | created by google.golang.org/grpc/internal/transport.newHTTP2Client
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x1a8c
2024-02-15 14:06:00 milvus-standalone  | 
2024-02-15 14:06:00 milvus-standalone  | goroutine 359 [IO wait]:
2024-02-15 14:06:00 milvus-standalone  | runtime.gopark(0x4000082ab8?, 0x2152df4?, 0xb8?, 0x2a?, 0x2152d2c?)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0x4000082a70 sp=0x4000082a50 pc=0x1b651d0
2024-02-15 14:06:00 milvus-standalone  | runtime.netpollblock(0x0?, 0xffffffff?, 0xff?)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/netpoll.go:527 +0x158 fp=0x4000082ab0 sp=0x4000082a70 pc=0x1b5d898
2024-02-15 14:06:00 milvus-standalone  | internal/poll.runtime_pollWait(0xffff4d0732b8, 0x72)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/netpoll.go:306 +0xa0 fp=0x4000082ae0 sp=0x4000082ab0 pc=0x1b93470
2024-02-15 14:06:00 milvus-standalone  | internal/poll.(*pollDesc).wait(0x40003c8000?, 0x40018e4000?, 0x0)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x4000082b10 sp=0x4000082ae0 pc=0x1c18b78
2024-02-15 14:06:00 milvus-standalone  | internal/poll.(*pollDesc).waitRead(...)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2024-02-15 14:06:00 milvus-standalone  | internal/poll.(*FD).Read(0x40003c8000, {0x40018e4000, 0x8000, 0x8000})
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x200 fp=0x4000082bb0 sp=0x4000082b10 pc=0x1c1a010
2024-02-15 14:06:00 milvus-standalone  | net.(*netFD).Read(0x40003c8000, {0x40018e4000?, 0x4000082cd8?, 0x1b351ac?})
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/net/fd_posix.go:55 +0x28 fp=0x4000082c00 sp=0x4000082bb0 pc=0x1dfd0c8
2024-02-15 14:06:00 milvus-standalone  | net.(*conn).Read(0x40001a1a10, {0x40018e4000?, 0x4001800340?, 0x4?})
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/net/net.go:183 +0x34 fp=0x4000082c50 sp=0x4000082c00 pc=0x1e10664
2024-02-15 14:06:00 milvus-standalone  | net.(*TCPConn).Read(0x4000082cf8?, {0x40018e4000?, 0x800000601?, 0x4000000000?})
2024-02-15 14:06:00 milvus-standalone  |        <autogenerated>:1 +0x2c fp=0x4000082c80 sp=0x4000082c50 pc=0x1e25fdc
2024-02-15 14:06:00 milvus-standalone  | bufio.(*Reader).Read(0x40019ec0c0, {0x4000dc09e0, 0x9, 0x1b355fc?})
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/bufio/bufio.go:237 +0x1d8 fp=0x4000082cc0 sp=0x4000082c80 pc=0x1cc2438
2024-02-15 14:06:00 milvus-standalone  | io.ReadAtLeast({0x516c6e0, 0x40019ec0c0}, {0x4000dc09e0, 0x9, 0x9}, 0x9)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/io/io.go:332 +0xa0 fp=0x4000082d10 sp=0x4000082cc0 pc=0x1bee560
2024-02-15 14:06:00 milvus-standalone  | io.ReadFull(...)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/io/io.go:351
2024-02-15 14:06:00 milvus-standalone  | golang.org/x/net/http2.readFrameHeader({0x4000dc09e0?, 0x9?, 0x4000082dc8?}, {0x516c6e0?, 0x40019ec0c0?})
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x58 fp=0x4000082d60 sp=0x4000082d10 pc=0x211f1b8
2024-02-15 14:06:00 milvus-standalone  | golang.org/x/net/http2.(*Framer).ReadFrame(0x4000dc09a0)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x80 fp=0x4000082e10 sp=0x4000082d60 pc=0x211fe50
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*http2Client).reader(0x4000f5e240, 0x0?)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x1b4 fp=0x4000082fb0 sp=0x4000082e10 pc=0x2157b94
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.newHTTP2Client.func11()
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2c fp=0x4000082fd0 sp=0x4000082fb0 pc=0x214e4dc
2024-02-15 14:06:00 milvus-standalone  | runtime.goexit()
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000082fd0 sp=0x4000082fd0 pc=0x1b9a5c4
2024-02-15 14:06:00 milvus-standalone  | created by google.golang.org/grpc/internal/transport.newHTTP2Client
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1638
2024-02-15 14:06:00 milvus-standalone  | 
2024-02-15 14:06:00 milvus-standalone  | goroutine 360 [runnable]:
2024-02-15 14:06:00 milvus-standalone  | runtime.gopark(0x4001837ef8?, 0x2?, 0x11?, 0x0?, 0x4001837ec4?)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0x4001837d70 sp=0x4001837d50 pc=0x1b651d0
2024-02-15 14:06:00 milvus-standalone  | runtime.selectgo(0x4001837ef8, 0x4001837ec0, 0x4001837ec8?, 0x0, 0x40001a1a10?, 0x1)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/select.go:327 +0x68c fp=0x4001837e80 sp=0x4001837d70 pc=0x1b7608c
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x400121e0f0, 0x1)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x14c fp=0x4001837f20 sp=0x4001837e80 pc=0x2144acc
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x4000fd80e0)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x7c fp=0x4001837f80 sp=0x4001837f20 pc=0x214532c
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x68 fp=0x4001837fd0 sp=0x4001837f80 pc=0x214e3e8
2024-02-15 14:06:00 milvus-standalone  | runtime.goexit()
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4001837fd0 sp=0x4001837fd0 pc=0x1b9a5c4
2024-02-15 14:06:00 milvus-standalone  | created by google.golang.org/grpc/internal/transport.newHTTP2Client
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x1a8c
2024-02-15 14:06:34 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:06:34.947Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:06:29.950Z","time spent":"4.997195294s","remote":"127.0.0.1:49842","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:07:10 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:07:10.074Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:07:05.074Z","time spent":"4.999448002s","remote":"127.0.0.1:49668","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:07:45 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:07:45.198Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:07:40.199Z","time spent":"4.998263086s","remote":"127.0.0.1:45374","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:08:20 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:08:20.319Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:08:15.315Z","time spent":"5.003715627s","remote":"127.0.0.1:52270","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:08:55 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:08:55.435Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:08:50.435Z","time spent":"4.999496419s","remote":"127.0.0.1:53226","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:09:30 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:09:30.550Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:09:25.541Z","time spent":"5.009804126s","remote":"127.0.0.1:50918","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:10:05 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:10:05.649Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:10:00.650Z","time spent":"4.998435752s","remote":"127.0.0.1:47196","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:10:40 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:10:40.773Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:10:35.772Z","time spent":"5.001399252s","remote":"127.0.0.1:54320","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:10:59 milvus-etcd        | {"level":"info","ts":"2024-02-15T22:10:59.920Z","caller":"v3compactor/revision.go:86","msg":"starting auto revision compaction","revision":14933,"revision-compaction-retention":1000}
2024-02-15 14:11:06 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:11:06.921Z","caller":"v3compactor/revision.go:101","msg":"failed auto revision compaction","revision":14933,"revision-compaction-retention":1000,"retry-interval":"5m0s","error":"etcdserver: request timed out"}

And the standalone container is not starting

Feb 15 '24 22:02 jacobmarks

I'm getting this issue in Docker,

...

2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x58 fp=0x40017bcd60 sp=0x40017bcd10 pc=0x211f1b8
2024-02-15 14:06:00 milvus-standalone  | golang.org/x/net/http2.(*Framer).ReadFrame(0x40011aa620)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x80 fp=0x40017bce10 sp=0x40017bcd60 pc=0x211fe50
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*http2Client).reader(0x4001658b40, 0x0?)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x1b4 fp=0x40017bcfb0 sp=0x40017bce10 pc=0x2157b94
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.newHTTP2Client.func11()
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2c fp=0x40017bcfd0 sp=0x40017bcfb0 pc=0x214e4dc
2024-02-15 14:06:00 milvus-standalone  | runtime.goexit()
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40017bcfd0 sp=0x40017bcfd0 pc=0x1b9a5c4
2024-02-15 14:06:00 milvus-standalone  | created by google.golang.org/grpc/internal/transport.newHTTP2Client
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1638
2024-02-15 14:06:00 milvus-standalone  | 
2024-02-15 14:06:00 milvus-standalone  | goroutine 418 [select]:
2024-02-15 14:06:00 milvus-standalone  | runtime.gopark(0x4000ccdef8?, 0x2?, 0x9?, 0x0?, 0x4000ccdec4?)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0x4000ccdd70 sp=0x4000ccdd50 pc=0x1b651d0
2024-02-15 14:06:00 milvus-standalone  | runtime.selectgo(0x4000ccdef8, 0x4000ccdec0, 0x4000ccdec8?, 0x0, 0x400059cef0?, 0x1)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/select.go:327 +0x68c fp=0x4000ccde80 sp=0x4000ccdd70 pc=0x1b7608c
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x400163b3b0, 0x1)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x14c fp=0x4000ccdf20 sp=0x4000ccde80 pc=0x2144acc
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x4000fe0230)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x7c fp=0x4000ccdf80 sp=0x4000ccdf20 pc=0x214532c
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x68 fp=0x4000ccdfd0 sp=0x4000ccdf80 pc=0x214e3e8
2024-02-15 14:06:00 milvus-standalone  | runtime.goexit()
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000ccdfd0 sp=0x4000ccdfd0 pc=0x1b9a5c4
2024-02-15 14:06:00 milvus-standalone  | created by google.golang.org/grpc/internal/transport.newHTTP2Client
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x1a8c
2024-02-15 14:06:00 milvus-standalone  | 
2024-02-15 14:06:00 milvus-standalone  | goroutine 359 [IO wait]:
2024-02-15 14:06:00 milvus-standalone  | runtime.gopark(0x4000082ab8?, 0x2152df4?, 0xb8?, 0x2a?, 0x2152d2c?)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0x4000082a70 sp=0x4000082a50 pc=0x1b651d0
2024-02-15 14:06:00 milvus-standalone  | runtime.netpollblock(0x0?, 0xffffffff?, 0xff?)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/netpoll.go:527 +0x158 fp=0x4000082ab0 sp=0x4000082a70 pc=0x1b5d898
2024-02-15 14:06:00 milvus-standalone  | internal/poll.runtime_pollWait(0xffff4d0732b8, 0x72)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/netpoll.go:306 +0xa0 fp=0x4000082ae0 sp=0x4000082ab0 pc=0x1b93470
2024-02-15 14:06:00 milvus-standalone  | internal/poll.(*pollDesc).wait(0x40003c8000?, 0x40018e4000?, 0x0)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x4000082b10 sp=0x4000082ae0 pc=0x1c18b78
2024-02-15 14:06:00 milvus-standalone  | internal/poll.(*pollDesc).waitRead(...)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2024-02-15 14:06:00 milvus-standalone  | internal/poll.(*FD).Read(0x40003c8000, {0x40018e4000, 0x8000, 0x8000})
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x200 fp=0x4000082bb0 sp=0x4000082b10 pc=0x1c1a010
2024-02-15 14:06:00 milvus-standalone  | net.(*netFD).Read(0x40003c8000, {0x40018e4000?, 0x4000082cd8?, 0x1b351ac?})
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/net/fd_posix.go:55 +0x28 fp=0x4000082c00 sp=0x4000082bb0 pc=0x1dfd0c8
2024-02-15 14:06:00 milvus-standalone  | net.(*conn).Read(0x40001a1a10, {0x40018e4000?, 0x4001800340?, 0x4?})
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/net/net.go:183 +0x34 fp=0x4000082c50 sp=0x4000082c00 pc=0x1e10664
2024-02-15 14:06:00 milvus-standalone  | net.(*TCPConn).Read(0x4000082cf8?, {0x40018e4000?, 0x800000601?, 0x4000000000?})
2024-02-15 14:06:00 milvus-standalone  |        <autogenerated>:1 +0x2c fp=0x4000082c80 sp=0x4000082c50 pc=0x1e25fdc
2024-02-15 14:06:00 milvus-standalone  | bufio.(*Reader).Read(0x40019ec0c0, {0x4000dc09e0, 0x9, 0x1b355fc?})
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/bufio/bufio.go:237 +0x1d8 fp=0x4000082cc0 sp=0x4000082c80 pc=0x1cc2438
2024-02-15 14:06:00 milvus-standalone  | io.ReadAtLeast({0x516c6e0, 0x40019ec0c0}, {0x4000dc09e0, 0x9, 0x9}, 0x9)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/io/io.go:332 +0xa0 fp=0x4000082d10 sp=0x4000082cc0 pc=0x1bee560
2024-02-15 14:06:00 milvus-standalone  | io.ReadFull(...)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/io/io.go:351
2024-02-15 14:06:00 milvus-standalone  | golang.org/x/net/http2.readFrameHeader({0x4000dc09e0?, 0x9?, 0x4000082dc8?}, {0x516c6e0?, 0x40019ec0c0?})
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x58 fp=0x4000082d60 sp=0x4000082d10 pc=0x211f1b8
2024-02-15 14:06:00 milvus-standalone  | golang.org/x/net/http2.(*Framer).ReadFrame(0x4000dc09a0)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x80 fp=0x4000082e10 sp=0x4000082d60 pc=0x211fe50
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*http2Client).reader(0x4000f5e240, 0x0?)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1592 +0x1b4 fp=0x4000082fb0 sp=0x4000082e10 pc=0x2157b94
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.newHTTP2Client.func11()
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x2c fp=0x4000082fd0 sp=0x4000082fb0 pc=0x214e4dc
2024-02-15 14:06:00 milvus-standalone  | runtime.goexit()
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000082fd0 sp=0x4000082fd0 pc=0x1b9a5c4
2024-02-15 14:06:00 milvus-standalone  | created by google.golang.org/grpc/internal/transport.newHTTP2Client
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:394 +0x1638
2024-02-15 14:06:00 milvus-standalone  | 
2024-02-15 14:06:00 milvus-standalone  | goroutine 360 [runnable]:
2024-02-15 14:06:00 milvus-standalone  | runtime.gopark(0x4001837ef8?, 0x2?, 0x11?, 0x0?, 0x4001837ec4?)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0x4001837d70 sp=0x4001837d50 pc=0x1b651d0
2024-02-15 14:06:00 milvus-standalone  | runtime.selectgo(0x4001837ef8, 0x4001837ec0, 0x4001837ec8?, 0x0, 0x40001a1a10?, 0x1)
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/select.go:327 +0x68c fp=0x4001837e80 sp=0x4001837d70 pc=0x1b7608c
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x400121e0f0, 0x1)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:417 +0x14c fp=0x4001837f20 sp=0x4001837e80 pc=0x2144acc
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x4000fd80e0)
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:549 +0x7c fp=0x4001837f80 sp=0x4001837f20 pc=0x214532c
2024-02-15 14:06:00 milvus-standalone  | google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:448 +0x68 fp=0x4001837fd0 sp=0x4001837f80 pc=0x214e3e8
2024-02-15 14:06:00 milvus-standalone  | runtime.goexit()
2024-02-15 14:06:00 milvus-standalone  |        /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4001837fd0 sp=0x4001837fd0 pc=0x1b9a5c4
2024-02-15 14:06:00 milvus-standalone  | created by google.golang.org/grpc/internal/transport.newHTTP2Client
2024-02-15 14:06:00 milvus-standalone  |        /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:446 +0x1a8c
2024-02-15 14:06:34 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:06:34.947Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:06:29.950Z","time spent":"4.997195294s","remote":"127.0.0.1:49842","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:07:10 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:07:10.074Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:07:05.074Z","time spent":"4.999448002s","remote":"127.0.0.1:49668","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:07:45 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:07:45.198Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:07:40.199Z","time spent":"4.998263086s","remote":"127.0.0.1:45374","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:08:20 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:08:20.319Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:08:15.315Z","time spent":"5.003715627s","remote":"127.0.0.1:52270","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:08:55 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:08:55.435Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:08:50.435Z","time spent":"4.999496419s","remote":"127.0.0.1:53226","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:09:30 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:09:30.550Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:09:25.541Z","time spent":"5.009804126s","remote":"127.0.0.1:50918","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:10:05 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:10:05.649Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:10:00.650Z","time spent":"4.998435752s","remote":"127.0.0.1:47196","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:10:40 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:10:40.773Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2024-02-15T22:10:35.772Z","time spent":"5.001399252s","remote":"127.0.0.1:54320","response type":"/etcdserverpb.Maintenance/Alarm","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
2024-02-15 14:10:59 milvus-etcd        | {"level":"info","ts":"2024-02-15T22:10:59.920Z","caller":"v3compactor/revision.go:86","msg":"starting auto revision compaction","revision":14933,"revision-compaction-retention":1000}
2024-02-15 14:11:06 milvus-etcd        | {"level":"warn","ts":"2024-02-15T22:11:06.921Z","caller":"v3compactor/revision.go:101","msg":"failed auto revision compaction","revision":14933,"revision-compaction-retention":1000,"retry-interval":"5m0s","error":"etcdserver: request timed out"}

And the standalone container is not starting

Could you collect logs from https://github.com/milvus-io/milvus/blob/a297baae9dda940444fe8813c07e29c1efc80001/deployments/export-log/README.md?plain=1#L33?

From the log you offered, your etcd response time is 5 seconds which might be the major issue. Please check your config and make sure you are deploying on high performance ssd

Feb 16 '24 02:02 xiaofan-luan

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

Jun 16 '24 04:06 stale[bot]

milvus milvus copied to clipboard

[Bug]: milvus standalone 2.3.2 panic

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

milvus
milvus copied to clipboard