kubeblocks
kubeblocks copied to clipboard
Published
20 hours ago •
apecloud
apecloud
[BUG]create milvus cluster failed with kbcli
kbcli cluster create testm --cluster-definition milvus-2.3.2
Info: --cluster-version is not specified, ClusterVersion is applied by default
Cluster testm created
➜ ~ k get pod
NAME READY STATUS RESTARTS AGE
testm-datanode-0 1/1 Running 0 13m
testm-etcd-0 1/1 Running 0 14m
testm-indexnode-0 1/1 Running 0 14m
testm-milvus-0 0/1 Running 6 (4m13s ago) 13m
testm-minio-0 1/1 Running 0 13m
testm-mixcoord-0 0/1 CrashLoopBackOff 3 (50s ago) 13m
testm-proxy-0 0/1 Running 2 (3m40s ago) 14m
testm-querynode-0 1/1 Running 0 14m
➜ ~ k logs testm-mixcoord-0
Defaulted container "mixcoord" out of: mixcoord, config (init)
2024/04/15 05:58:47 missing source yaml
2024/04/15 05:58:47 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
__ _________ _ ____ ______
/ |/ / _/ /| | / / / / / __/
/ /|_/ // // /_| |/ / /_/ /\ \
/_/ /_/___/____/___/\____/___/
Welcome to use Milvus!
Version: v2.3.2-dev
Built: Thu Oct 26 09:30:40 UTC 2023
GitCommit: 8b453454d
GoVersion: go version go1.20.7 linux/amd64
open pid file: /run/milvus/mixture.pid
lock pid file: /run/milvus/mixture.pid
[2024/04/15 05:58:47.391 +00:00] [INFO] [roles/roles.go:304] ["starting running Milvus components"]
[2024/04/15 05:58:47.391 +00:00] [INFO] [roles/roles.go:167] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2024/04/15 05:58:47.395 +00:00] [DEBUG] [config/etcd_source.go:49] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"UseSSL\":false,\"Endpoints\":[\"testm-etcd-headless.default.svc.cluster.local:2379\"],\"KeyPrefix\":\"testm\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"]
[2024/04/15 05:58:47.395 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[testm-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/04/15 05:58:47.396 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=testm/config] [endpoints="[testm-etcd-headless.default.svc.cluster.local:2379]"]
[2024/04/15 05:58:47.396 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=FileSource]
[2024/04/15 05:58:47.427 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/04/15 05:58:47.427 +00:00] [INFO] [roles/roles.go:253] [setupPrometheusHTTPServer]
[2024/04/15 05:58:47.427 +00:00] [INFO] [rootcoord/root_coord.go:155] ["update rootcoord state"] [state=Abnormal]
[2024/04/15 05:58:47.427 +00:00] [DEBUG] [rootcoord/service.go:166] ["init params done.."]
[2024/04/15 05:58:47.427 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[testm-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/04/15 05:58:47.428 +00:00] [DEBUG] [rootcoord/service.go:183] ["etcd connect done ..."]
[2024/04/15 05:58:47.428 +00:00] [DEBUG] [rootcoord/service.go:243] ["start grpc "] [port=53100]
[2024/04/15 05:58:47.428 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=EtcdSource]
[2024/04/15 05:58:47.428 +00:00] [INFO] [config/refresher.go:66] ["start refreshing configurations"] [source=FileSource]
[2024/04/15 05:58:47.428 +00:00] [INFO] [http/server.go:77] ["management listen"] [addr=:9091]
[2024/04/15 05:58:47.428 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[testm-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/04/15 05:58:47.429 +00:00] [DEBUG] [querycoord/service.go:215] [network] [port=19531]
[2024/04/15 05:58:47.429 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[testm-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/04/15 05:58:47.430 +00:00] [DEBUG] [datacoord/service.go:147] ["network port"] [port=13333]
[2024/04/15 05:58:47.430 +00:00] [INFO] [components/index_coord.go:38] ["IndexCoord running ..."]
[2024/04/15 05:58:47.529 +00:00] [DEBUG] [rootcoord/service.go:200] ["grpc init done ..."]
[2024/04/15 05:58:47.529 +00:00] [DEBUG] [rootcoord/service.go:203] ["RootCoord start to create DataCoord client"]
[2024/04/15 05:58:47.529 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"]
[2024/04/15 05:58:47.529 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"]
[2024/04/15 05:58:47.530 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"]
[2024/04/15 05:58:47.530 +00:00] [DEBUG] [querycoord/service.go:165] ["QueryCoord try to wait for RootCoord ready"]
[2024/04/15 05:58:47.530 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"]
[2024/04/15 05:58:47.530 +00:00] [DEBUG] [rootcoord/service.go:212] ["RootCoord start to create QueryCoord client"]
[2024/04/15 05:58:47.530 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"]
[2024/04/15 05:58:47.530 +00:00] [INFO] [dependency/factory.go:84] ["try to init mq"] [standalone=false] [mqType=pulsar]
[2024/04/15 05:58:47.530 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"]
[2024/04/15 05:58:47.531 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=10.40.1.194:53100]
[2024/04/15 05:58:47.531 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=10.40.1.194:53100] [serverID=16]
[2024/04/15 05:58:47.531 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"]
[2024/04/15 05:58:47.531 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"]
[2024/04/15 05:58:47.531 +00:00] [INFO] [dependency/factory.go:84] ["try to init mq"] [standalone=false] [mqType=pulsar]
[2024/04/15 05:58:47.532 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"]
[2024/04/15 05:58:47.532 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"]
[2024/04/15 05:58:47.534 +00:00] [DEBUG] [sessionutil/session_util.go:292] [getServerID] [reuse=true]
[2024/04/15 05:58:47.534 +00:00] [DEBUG] [querycoord/service.go:175] ["QueryCoord report RootCoord ready"]
[2024/04/15 05:58:47.534 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"]
[2024/04/15 05:58:47.535 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"]
[2024/04/15 05:58:47.535 +00:00] [DEBUG] [querycoord/service.go:186] ["QueryCoord try to wait for DataCoord ready"]
[2024/04/15 05:58:47.536 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=datacoord] [key=datacoord] [address=10.40.1.194:13333]
[2024/04/15 05:58:47.537 +00:00] [DEBUG] [sessionutil/session_util.go:350] ["Session get serverID success"] [key=id] [ServerId=18]
[2024/04/15 05:58:47.537 +00:00] [INFO] [sessionutil/session_util.go:266] ["start server"] [name=indexcoord] [address=10.40.2.169:13333] [id=18]
[2024/04/15 05:58:47.537 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"]
[2024/04/15 05:58:47.538 +00:00] [DEBUG] [sessionutil/session_util.go:292] [getServerID] [reuse=true]
[2024/04/15 05:58:47.538 +00:00] [INFO] [sessionutil/session_util.go:266] ["start server"] [name=rootcoord] [address=10.40.2.169:53100] [id=18]
[2024/04/15 05:58:47.538 +00:00] [INFO] [rootcoord/root_coord.go:155] ["update rootcoord state"] [state=Initializing]
[2024/04/15 05:58:47.538 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"]
[2024/04/15 05:58:47.539 +00:00] [DEBUG] [querycoord/service.go:195] ["QueryCoord report DataCoord ready"]
[2024/04/15 05:58:47.539 +00:00] [INFO] [querycoordv2/server.go:177] ["QueryCoord start init"] [meta-root-path=testm/meta] [address=10.40.2.169:19531]
[2024/04/15 05:58:47.539 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"]
[2024/04/15 05:58:47.541 +00:00] [DEBUG] [sessionutil/session_util.go:292] [getServerID] [reuse=true]
[2024/04/15 05:58:47.541 +00:00] [INFO] [sessionutil/session_util.go:266] ["start server"] [name=datacoord] [address=10.40.2.169:13333] [id=18]
[2024/04/15 05:58:47.541 +00:00] [DEBUG] [sessionutil/session_util.go:234] ["Session try to connect to etcd"]
[2024/04/15 05:58:47.542 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"]
[2024/04/15 05:58:47.543 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2024/04/15 05:57:17.377 +00:00] [save=2024/04/15 05:58:50.538 +00:00] [next=2024/04/15 05:58:47.538 +00:00]
[2024/04/15 05:58:47.543 +00:00] [INFO] [rootcoord/root_coord.go:398] ["id allocator initialized"] [root_path=testm/kv] [sub_path=gid] [key=idTimestamp]
[2024/04/15 05:58:47.543 +00:00] [DEBUG] [sessionutil/session_util.go:249] ["Session connect to etcd success"]
[2024/04/15 05:58:47.546 +00:00] [DEBUG] [sessionutil/session_util.go:292] [getServerID] [reuse=true]
[2024/04/15 05:58:47.546 +00:00] [INFO] [sessionutil/session_util.go:266] ["start server"] [name=querycoord] [address=10.40.2.169:19531] [id=18]
[2024/04/15 05:58:47.546 +00:00] [INFO] [querycoordv2/server.go:209] [QueryCoord] [State=Initializing]
[2024/04/15 05:58:47.546 +00:00] [INFO] [querycoordv2/server.go:213] ["query coordinator connecting to etcd."]
[2024/04/15 05:58:47.546 +00:00] [INFO] [querycoordv2/server.go:223] ["query coordinator successfully connected to etcd."]
[2024/04/15 05:58:47.548 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2024/04/15 05:58:50.465 +00:00] [save=2024/04/15 05:58:53.466 +00:00] [next=2024/04/15 05:58:50.466 +00:00]
[2024/04/15 05:58:47.548 +00:00] [INFO] [rootcoord/root_coord.go:422] ["tso allocator initialized"] [root_path=testm/kv] [sub_path=gid] [key=idTimestamp]
[2024/04/15 05:58:47.548 +00:00] [INFO] [rootcoord/root_coord.go:341] ["Using etcd as meta storage."]
[2024/04/15 05:58:47.548 +00:00] [DEBUG] [rootcoord/suffix_snapshot.go:555] ["suffix snapshot GC goroutine start!"]
[2024/04/15 05:58:47.550 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2024/04/15 05:57:40.572 +00:00] [save=2024/04/15 05:58:50.547 +00:00] [next=2024/04/15 05:58:47.547 +00:00]
[2024/04/15 05:58:47.550 +00:00] [INFO] [querycoordv2/server.go:312] ["init meta"]
[2024/04/15 05:58:47.550 +00:00] [INFO] [querycoordv2/server.go:321] ["recover meta..."]
[2024/04/15 05:58:47.550 +00:00] [INFO] [rootcoord/meta_table.go:150] ["recover databases"] ["num of dbs"=1]
[2024/04/15 05:58:47.551 +00:00] [INFO] [querycoordv2/server.go:322] ["recover collections and partitions from kv store"] [traceID=1713160727551476382]
[2024/04/15 05:58:47.551 +00:00] [INFO] [querycoordv2/server.go:328] ["recovering collections..."] [collections="[]"]
[2024/04/15 05:58:47.553 +00:00] [INFO] [meta/resource_manager.go:675] ["Recover resource group"] [rgName=__default_resource_group] [nodes="[1,4,16]"] [capacity=1000000]
[2024/04/15 05:58:47.553 +00:00] [INFO] [querycoordv2/server.go:353] ["QueryCoord server initMeta done"] [duration=3.170149ms]
[2024/04/15 05:58:47.553 +00:00] [INFO] [querycoordv2/server.go:245] ["init session"]
[2024/04/15 05:58:47.553 +00:00] [INFO] [querycoordv2/server.go:249] ["init schedulers"]
[2024/04/15 05:58:47.553 +00:00] [INFO] [querycoordv2/server.go:262] ["init dist controller"]
[2024/04/15 05:58:47.553 +00:00] [INFO] [querycoordv2/server.go:272] ["init all available balancer"]
[2024/04/15 05:58:47.553 +00:00] [INFO] [querycoordv2/server.go:281] ["use config balancer"] [balancer=ScoreBasedBalancer]
[2024/04/15 05:58:47.553 +00:00] [INFO] [querycoordv2/server.go:288] ["init checker controller"]
[2024/04/15 05:58:47.553 +00:00] [INFO] [querycoordv2/server.go:358] ["init observers"]
[2024/04/15 05:58:47.553 +00:00] [INFO] [querycoordv2/server.go:305] ["QueryCoord init success"]
[2024/04/15 05:58:47.553 +00:00] [DEBUG] [querycoord/service.go:103] ["QueryCoord init done ..."]
[2024/04/15 05:58:47.553 +00:00] [DEBUG] [sessionutil/session_util.go:413] ["service begin to register to etcd"] [serverName=querycoord] [ServerID=18]
[2024/04/15 05:58:47.559 +00:00] [INFO] [rootcoord/meta_table.go:188] ["recover collections from db"] [collection_num=0] [partition_num=0]
[2024/04/15 05:58:47.559 +00:00] [ERROR] [sessionutil/session_util.go:455] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: querycoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276\ngithub.com/milvus-io/milvus/internal/querycoordv2.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/server.go:140\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:264\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:105\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"]
[2024/04/15 05:58:47.561 +00:00] [INFO] [rootcoord/meta_table.go:204] ["RootCoord meta table reload done"] [duration=13.180619ms]
[2024/04/15 05:58:47.561 +00:00] [INFO] [msgstream/mq_factory.go:103] ["Msg Stream state"] [can_produce=true]
time="2024-04-15T05:58:47Z" level=info msg="Connecting to broker" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
[2024/04/15 05:58:47.562 +00:00] [INFO] [storage/minio_chunk_manager.go:86] ["minio chunk manager init success."] [bucketname=testm] [root=files]
[2024/04/15 05:58:47.563 +00:00] [INFO] [datacoord/server.go:552] ["data coordinator connecting to metadata store"] [metaType=etcd]
[2024/04/15 05:58:47.563 +00:00] [INFO] [datacoord/server.go:560] ["data coordinator successfully connected to metadata store"] [metaType=etcd]
[2024/04/15 05:58:47.565 +00:00] [INFO] [datacoord/meta.go:171] ["DataCoord meta reloadFromKV done"] [duration=2.34307ms]
[2024/04/15 05:58:47.566 +00:00] [INFO] [datacoord/channel_store.go:171] ["channel store reload done"] [duration=440.33µs]
[2024/04/15 05:58:47.566 +00:00] [DEBUG] [sessionutil/session_util.go:596] ["SessionUtil GetSessions "] [prefix=datanode] [key=datanode-16] [address=10.40.1.194:21124]
[2024/04/15 05:58:47.566 +00:00] [DEBUG] [sessionutil/session_util.go:596] ["SessionUtil GetSessions "] [prefix=datanode] [key=datanode-5] [address=10.40.0.197:21124]
[2024/04/15 05:58:47.566 +00:00] [INFO] [datacoord/server.go:485] ["DataCoord success to get DataNode sessions"] [sessions="{\"datanode-16\":{\"ServerID\":16,\"ServerName\":\"datanode\",\"Address\":\"10.40.1.194:21124\",\"TriggerKill\":true,\"Version\":\"2.3.2\",\"IndexEngineVersion\":{},\"LeaseID\":752258231731963091,\"HostName\":\"testm-milvus-0\"},\"datanode-5\":{\"ServerID\":5,\"ServerName\":\"datanode\",\"Address\":\"10.40.0.197:21124\",\"TriggerKill\":true,\"Version\":\"2.3.2\",\"IndexEngineVersion\":{},\"LeaseID\":752258231731959858,\"HostName\":\"testm-datanode-0\"}}"]
[2024/04/15 05:58:47.566 +00:00] [INFO] [datacoord/channel_manager.go:348] ["register node with no reassignment"] ["registered node"=16]
[2024/04/15 05:58:47.566 +00:00] [INFO] [datacoord/channel_manager.go:348] ["register node with no reassignment"] ["registered node"=5]
[2024/04/15 05:58:47.566 +00:00] [INFO] [datacoord/channel_manager.go:171] ["starting etcd states checker"]
[2024/04/15 05:58:47.566 +00:00] [INFO] [datacoord/channel_manager.go:176] ["starting background balance checker"]
[2024/04/15 05:58:47.566 +00:00] [INFO] [datacoord/channel_manager.go:179] ["cluster start up"] [nodes="[16,5]"] [oNodes="[]"] ["old onlines"="[]"] ["new onlines"="[16,5]"] [offLines="[]"]
[2024/04/15 05:58:47.567 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-16] [address=10.40.1.194:21121]
[2024/04/15 05:58:47.567 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-3] [address=10.40.2.168:21121]
[2024/04/15 05:58:47.567 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=16] ["node address"=10.40.1.194:21121]
[2024/04/15 05:58:47.567 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=16]
[2024/04/15 05:58:47.568 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=16] ["IndexNode num"=1]
[2024/04/15 05:58:47.568 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=3] ["node address"=10.40.2.168:21121]
[2024/04/15 05:58:47.568 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=3]
[2024/04/15 05:58:47.568 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=3] ["IndexNode num"=2]
[2024/04/15 05:58:47.568 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=querynode] [key=querynode-1] [address=10.40.1.193:21123]
[2024/04/15 05:58:47.568 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=querynode] [key=querynode-16] [address=10.40.1.194:21123]
[2024/04/15 05:58:47.568 +00:00] [INFO] [datacoord/index_engine_version_manager.go:55] ["addOrUpdate version"] [nodeId=16] [minimal=0] [current=1]
[2024/04/15 05:58:47.568 +00:00] [INFO] [datacoord/index_engine_version_manager.go:55] ["addOrUpdate version"] [nodeId=1] [minimal=0] [current=1]
[2024/04/15 05:58:47.569 +00:00] [INFO] [datacoord/garbage_collector.go:67] ["GC with option"] [enabled=true] [interval=1h0m0s] [missingTolerance=1h0m0s] [dropTolerance=3h0m0s]
[2024/04/15 05:58:47.569 +00:00] [DEBUG] [datacoord/service.go:252] ["DataCoord init done ..."]
[2024/04/15 05:58:47.569 +00:00] [DEBUG] [sessionutil/session_util.go:413] ["service begin to register to etcd"] [serverName=indexcoord] [ServerID=18]
[2024/04/15 05:58:47.573 +00:00] [ERROR] [sessionutil/session_util.go:455] ["retry func failed"] ["retry time"=0] [error="function CompareAndSwap error for compare is false for key: indexcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:250\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:205\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:254\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:50\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"]
time="2024-04-15T05:58:47Z" level=warning msg="Failed to connect to broker." error="dial tcp: lookup testm-pulsar-headless.default.svc.cluster.local: no such host" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
time="2024-04-15T05:58:47Z" level=info msg="Connecting to broker" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
time="2024-04-15T05:58:47Z" level=warning msg="Failed to connect to broker." error="dial tcp: lookup testm-pulsar-headless.default.svc.cluster.local: no such host" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
time="2024-04-15T05:58:48Z" level=info msg="Connecting to broker" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
time="2024-04-15T05:58:48Z" level=warning msg="Failed to connect to broker." error="dial tcp: lookup testm-pulsar-headless.default.svc.cluster.local: no such host" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
time="2024-04-15T05:58:49Z" level=info msg="Connecting to broker" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
time="2024-04-15T05:58:49Z" level=warning msg="Failed to connect to broker." error="dial tcp: lookup testm-pulsar-headless.default.svc.cluster.local: no such host" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
[2024/04/15 05:58:50.580 +00:00] [INFO] [sessionutil/session_util.go:442] ["put session key into etcd"] [key=testm/meta/session/querycoord] [value="{\"ServerID\":18,\"ServerName\":\"querycoord\",\"Address\":\"10.40.2.169:19531\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.3.2\",\"IndexEngineVersion\":{},\"LeaseID\":752258231731963424,\"HostName\":\"testm-mixcoord-0\"}"]
[2024/04/15 05:58:50.580 +00:00] [INFO] [sessionutil/session_util.go:452] ["Service registered successfully"] [ServerName=querycoord] [serverID=18]
[2024/04/15 05:58:50.582 +00:00] [INFO] [querycoordv2/server.go:403] ["start watcher..."]
[2024/04/15 05:58:50.583 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=querynode] [key=querynode-1] [address=10.40.1.193:21123]
[2024/04/15 05:58:50.583 +00:00] [DEBUG] [sessionutil/session_util.go:566] ["SessionUtil GetSessions"] [prefix=querynode] [key=querynode-16] [address=10.40.1.194:21123]
[2024/04/15 05:58:50.583 +00:00] [INFO] [task/scheduler.go:230] ["add executor for new QueryNode"] [nodeID=1]
[2024/04/15 05:58:50.583 +00:00] [INFO] [task/scheduler.go:230] ["add executor for new QueryNode"] [nodeID=16]
[2024/04/15 05:58:50.583 +00:00] [INFO] [meta/resource_manager.go:445] ["HandleNodeUp: node already assign to resource group"] [rgName=__default_resource_group] [node=1]
[2024/04/15 05:58:50.583 +00:00] [INFO] [querycoordv2/server.go:712] ["HandleNodeUp: assign node to resource group"] [nodeID=1] [resourceGroup=__default_resource_group]
[2024/04/15 05:58:50.583 +00:00] [INFO] [meta/resource_manager.go:445] ["HandleNodeUp: node already assign to resource group"] [rgName=__default_resource_group] [node=16]
[2024/04/15 05:58:50.583 +00:00] [INFO] [querycoordv2/server.go:712] ["HandleNodeUp: assign node to resource group"] [nodeID=16] [resourceGroup=__default_resource_group]
[2024/04/15 05:58:50.584 +00:00] [INFO] [runtime/asm_amd64.s:1598] ["start dist handler"] [nodeID=1]
[2024/04/15 05:58:50.584 +00:00] [INFO] [runtime/asm_amd64.s:1598] ["start dist handler"] [nodeID=16]
[2024/04/15 05:58:50.587 +00:00] [WARN] [dist/dist_handler.go:100] ["node last heart beat time lag too behind"] [now=2024/04/15 05:58:50.587 +00:00] [lastHeartBeatTime=1970/01/01 00:00:00.000 +00:00] [nodeID=16]
[2024/04/15 05:58:50.588 +00:00] [WARN] [dist/dist_handler.go:100] ["node last heart beat time lag too behind"] [now=2024/04/15 05:58:50.588 +00:00] [lastHeartBeatTime=1970/01/01 00:00:00.000 +00:00] [nodeID=1]
[2024/04/15 05:58:50.588 +00:00] [INFO] [querycoordv2/server.go:438] ["start cluster..."]
[2024/04/15 05:58:50.588 +00:00] [INFO] [querycoordv2/server.go:441] ["start observers..."]
[2024/04/15 05:58:50.588 +00:00] [INFO] [observers/target_observer.go:113] ["Start update next target loop"]
[2024/04/15 05:58:50.588 +00:00] [INFO] [observers/target_observer.go:127] ["target observer init done"]
[2024/04/15 05:58:50.588 +00:00] [INFO] [querycoordv2/server.go:448] ["start task scheduler..."]
[2024/04/15 05:58:50.588 +00:00] [INFO] [querycoordv2/server.go:451] ["start checker controller..."]
[2024/04/15 05:58:50.588 +00:00] [INFO] [querycoordv2/server.go:454] ["start job scheduler..."]
[2024/04/15 05:58:50.588 +00:00] [INFO] [observers/replica_observer.go:68] ["Start check replica loop"]
[2024/04/15 05:58:50.588 +00:00] [INFO] [observers/resource_observer.go:66] ["Start check resource group loop"]
[2024/04/15 05:58:50.588 +00:00] [INFO] [sessionutil/session_util.go:1167] ["save server info into file"] [content="querycoord-18\n"] [filePath=/tmp/milvus/server_id_7]
[2024/04/15 05:58:50.588 +00:00] [INFO] [querycoordv2/server.go:397] ["QueryCoord started"]
[2024/04/15 05:58:50.588 +00:00] [DEBUG] [querycoord/service.go:108] ["QueryCoord start done ..."]
[2024/04/15 05:58:50.588 +00:00] [DEBUG] [components/query_coord.go:57] ["QueryCoord successfully started"]
[2024/04/15 05:58:50.595 +00:00] [ERROR] [sessionutil/session_util.go:455] ["retry func failed"] ["retry time"=4] [error="function CompareAndSwap error for compare is false for key: indexcoord"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.(*Session).registerService\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:455\ngithub.com/milvus-io/milvus/internal/util/sessionutil.(*Session).Register\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:276\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Register\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:250\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).start\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:205\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:254\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:50\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"]
[2024/04/15 05:58:50.648 +00:00] [INFO] [sessionutil/session_util.go:859] ["register session success"] [role=querycoord] [key=testm/meta/session/querycoord]
time="2024-04-15T05:58:51Z" level=info msg="Connecting to broker" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
time="2024-04-15T05:58:51Z" level=warning msg="Failed to connect to broker." error="dial tcp: lookup testm-pulsar-headless.default.svc.cluster.local: no such host" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
[2024/04/15 05:58:52.429 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=testm/config] [endpoints="[testm-etcd-headless.default.svc.cluster.local:2379]"]
[2024/04/15 05:58:52.442 +00:00] [DEBUG] [querycoord/service.go:380] ["get metrics request received"] [metricType="{\"metric_type\":\"system_info\"}"]
time="2024-04-15T05:58:54Z" level=info msg="Connecting to broker" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
time="2024-04-15T05:58:54Z" level=warning msg="Failed to connect to broker." error="dial tcp: lookup testm-pulsar-headless.default.svc.cluster.local: no such host" remote_addr="pulsar://testm-pulsar-headless.default.svc.cluster.local:6650"
[2024/04/15 05:58:57.429 +00:00] [DEBUG] [config/etcd_source.go:141] ["etcd refreshConfigurations"] [prefix=testm/config] [endpoints="[testm-etcd-headless.default.svc.cluster.local:2379]"]
[2024/04/15 05:58:59.086 +00:00] [WARN] [grpcclient/client.go:422] ["call received grpc error"] [clientRole=querynode-16] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""]
[2024/04/15 05:58:59.086 +00:00] [WARN] [retry/retry.go:44] ["start to reset connection because of specific reasons"] [client_role=querynode-16] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""]
[2024/04/15 05:58:59.087 +00:00] [WARN] [grpcclient/client.go:429] ["fail to get grpc client in the retry state"] [client_role=querynode-16] [error="failed to connect 10.40.1.194:21123, reason: connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""]
[2024/04/15 05:58:59.087 +00:00] [ERROR] [grpcclient/client.go:411] ["retry func failed"] ["retry time"=0] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505\ngithub.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:75\ngithub.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).GetDataDistribution\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:294\ngithub.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetDataDistribution.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:213\ngithub.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:276\ngithub.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetDataDistribution\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:208\ngithub.com/milvus-io/milvus/internal/querycoordv2/dist.(*distHandler).getDistribution\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/dist/dist_handler.go:230\ngithub.com/milvus-io/milvus/internal/querycoordv2/dist.(*distHandler).start\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/dist/dist_handler.go:75"]
[2024/04/15 05:58:59.288 +00:00] [WARN] [retry/retry.go:44] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=querynode-16] [error="empty grpc client: failed to connect 10.40.1.194:21123, reason: connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""]
[2024/04/15 05:58:59.290 +00:00] [WARN] [grpcclient/client.go:415] ["fail to get grpc client in the retry state"] [client_role=querynode-16] [error="failed to connect 10.40.1.194:21123, reason: connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""]
[2024/04/15 05:58:59.586 +00:00] [WARN] [grpcclient/client.go:489] ["fail to get grpc client"] [client_role=querynode-16] [error="failed to connect 10.40.1.194:21123, reason: connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""]
[2024/04/15 05:58:59.586 +00:00] [WARN] [retry/retry.go:44] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=querynode-16] [error="empty grpc client: failed to connect 10.40.1.194:21123, reason: connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""]
[2024/04/15 05:58:59.587 +00:00] [WARN] [grpcclient/client.go:415] ["fail to get grpc client in the retry state"] [client_role=querynode-16] [error="failed to connect 10.40.1.194:21123, reason: connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""]
[2024/04/15 05:58:59.587 +00:00] [ERROR] [grpcclient/client.go:411] ["retry func failed"] ["retry time"=0] [error="empty grpc client: failed to connect 10.40.1.194:21123, reason: connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:411\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:489\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505\ngithub.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:75\ngithub.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:89\ngithub.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:257\ngithub.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:276\ngithub.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:256\ngithub.com/milvus-io/milvus/internal/querycoordv2.(*Server).checkNodeHealth.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/services.go:991\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2024/04/15 05:58:59.685 +00:00] [WARN] [grpcclient/client.go:505] ["ClientBase Call grpc call get error"] [role=querynode-16] [address=] [error="stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:491 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:505 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:75 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:89 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:257 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:276 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:256 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/services.go:991 github.com/milvus-io/milvus/internal/querycoordv2.(*Server).checkNodeHealth.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1: attempt #0: empty grpc client: failed to connect 10.40.1.194:21123, reason: connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\": context deadline exceeded"]
[2024/04/15 05:58:59.685 +00:00] [WARN] [querycoordv2/server.go:667] ["unhealthy node exist, node up will be delayed"] [delayedNodeUpEvents=0] [unhealthyNodeNum=0] [unhealthyReason="[]"]
[2024/04/15 05:58:59.691 +00:00] [WARN] [retry/retry.go:44] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=querynode-16] [error="empty grpc client: failed to connect 10.40.1.194:21123, reason: connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""]
[2024/04/15 05:58:59.692 +00:00] [WARN] [grpcclient/client.go:415] ["fail to get grpc client in the retry state"] [client_role=querynode-16] [error="failed to connect 10.40.1.194:21123, reason: connection error: desc = \"transport: error while dialing: dial tcp 10.40.1.194:21123: connect: connection refused\""]
➜ ~k logs testm-milvus-0
...goroutine 1050 [IO wait]:
runtime.gopark(0x2?, 0xb?, 0x0?, 0x0?, 0x53?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0022f9aa0 sp=0xc0022f9a80 pc=0x17870d6
runtime.netpollblock(0x1822145?, 0x174b76f?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0022f9ad8 sp=0xc0022f9aa0 pc=0x177f2f7
internal/poll.runtime_pollWait(0x7e9de8bf7578, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0022f9af8 sp=0xc0022f9ad8 pc=0x17b67a9
internal/poll.(*pollDesc).wait(0xc0023cc200?, 0xc0023f8000?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0022f9b20 sp=0xc0022f9af8 pc=0x1841772
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0023cc200, {0xc0023f8000, 0x8000, 0x8000})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0022f9bb8 sp=0xc0022f9b20 pc=0x1842cd9
net.(*netFD).Read(0xc0023cc200, {0xc0023f8000?, 0x60100000000?, 0x8?})
/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0022f9c00 sp=0xc0022f9bb8 pc=0x1a69a29
net.(*conn).Read(0xc0020871e0, {0xc0023f8000?, 0x17871c0?, 0xc0022f9cc8?})
/usr/local/go/src/net/net.go:183 +0x45 fp=0xc0022f9c48 sp=0xc0022f9c00 pc=0x1a7ea05
net.(*TCPConn).Read(0x18?, {0xc0023f8000?, 0x42e2ae0?, 0x801000601?})
<autogenerated>:1 +0x29 fp=0xc0022f9c78 sp=0xc0022f9c48 pc=0x1a95569
bufio.(*Reader).Read(0xc00236bf20, {0xc0022e7000, 0x9, 0xc0022f9cf0?})
/usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0022f9cb0 sp=0xc0022f9c78 pc=0x18f7e1b
io.ReadAtLeast({0x4bd2e40, 0xc00236bf20}, {0xc0022e7000, 0x9, 0x9}, 0x9)
/usr/local/go/src/io/io.go:332 +0x9a fp=0xc0022f9cf8 sp=0xc0022f9cb0 pc=0x18158fa
io.ReadFull(...)
/usr/local/go/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0022e7000?, 0x9?, 0x4bd85e0?}, {0x4bd2e40?, 0xc00236bf20?})
/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e fp=0xc0022f9d48 sp=0xc0022f9cf8 pc=0x1ddd14e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0022e6fc0)
/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95 fp=0xc0022f9df8 sp=0xc0022f9d48 pc=0x1dddef5
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0023836c0, 0x0?, 0xc00217b738?)
/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:637 +0x174 fp=0xc0022f9f08 sp=0xc0022f9df8 pc=0x1e20674
google.golang.org/grpc.(*Server).serveStreams(0xc00025dc20, {0x4c10380?, 0xc0023836c0})
/go/pkg/mod/google.golang.org/[email protected]/server.go:950 +0x189 fp=0xc0022f9f80 sp=0xc0022f9f08 pc=0x1ea2529
google.golang.org/grpc.(*Server).handleRawConn.func1()
/go/pkg/mod/google.golang.org/[email protected]/server.go:892 +0x46 fp=0xc0022f9fe0 sp=0xc0022f9f80 pc=0x1ea1da6
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0022f9fe8 sp=0xc0022f9fe0 pc=0x17bd9a1
created by google.golang.org/grpc.(*Server).handleRawConn
/go/pkg/mod/google.golang.org/[email protected]/server.go:891 +0x185
➜ ~ k logs testm-milvus-0 >milvus.txt Defaulted container "milvus" out of: milvus, milvus-init (init) milvus.txt
milvus cluster can be created by kubectl following the guide https://github.com/apecloud/kubeblocks/blob/main/examples/milvus/README.md
This issue has been marked as stale because it has been open for 30 days with no activity
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}