milvus [Bug]: query node crashed and search timeout

[Bug]: query node crashed and search timeout

Open yhmo opened this issue 2 years ago • 16 comments

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster): dev build of 20220704
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

7 query nodes deployed, continually search, after a periods of time, 4 of the nodes crashed and search timeout.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

Contact @yhmo to get full log.

No response

Anything else?

No response

Jul 13 '22 07:07 yhmo

partial log of query node 001:

[2022/07/07 18:05:22.327 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:22.338 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=11] [2022/07/07 18:05:22.338 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=11] [addr=] [2022/07/07 18:05:22.338 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=11] [2022/07/07 18:05:22.338 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=11] [addr=] [2022/07/07 18:05:22.404 +08:00] [INFO] [impl.go:229] ["loadSegmentsTask Enqueue done"] [collectionID=434328479553028097] [segmentIDs="[434421958212583425]"] [nodeID=26] [2022/07/07 18:05:22.404 +08:00] [INFO] [task.go:507] ["LoadSegmentTask PreExecute start"] [msgID=0] [2022/07/07 18:05:22.404 +08:00] [INFO] [task.go:533] ["LoadSegmentTask PreExecute done"] [msgID=0] [2022/07/07 18:05:22.404 +08:00] [INFO] [task.go:539] ["LoadSegmentTask Execute start"] [msgID=0] [2022/07/07 18:05:22.404 +08:00] [INFO] [segment_loader.go:86] ["segmentLoader start loading..."] [collectionID=434328479553028097] [segmentNum=1] [segmentType=Sealed] [2022/07/07 18:05:22.405 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434405864160362497] [segmentID=434421958212583425] [segmentType=Sealed] [2022/07/07 18:05:22.405 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434405864160362497] [segmentID=434421958212583425] [segmentType=Sealed] [2022/07/07 18:05:22.595 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434421958212583425] [fieldID=102] [2022/07/07 18:05:22.595 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434421958212583425] [fieldID=102] [2022/07/07 18:05:22.609 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=33716] [segmentID=434421958212583425] [2022/07/07 18:05:22.625 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=33716] [segmentID=434421958212583425] [2022/07/07 18:05:22.625 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=33716] [segmentID=434421958212583425] [2022/07/07 18:05:22.625 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=33716] [segmentID=434421958212583425] [2022/07/07 18:05:22.625 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434421958212583425] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:22.771 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434421958212583425] [2022/07/07 18:05:22.804 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434405864160362497] [segmentID=434421958212583425] [segmentType=Sealed] [2022/07/07 18:05:22.804 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_20_434328479553028097v2" msgID:"\010\250\343\021\020\303\372\001\030\000 \000" msgGroup:"by-dev-dataNode-22-434328479553028097" timestamp:434421908746010629 "] ["msg id"=CKjjERDD+gEYACAA] [2022/07/07 18:05:22.809 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_20] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:22.818 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_20] [MessageID={}] [2022/07/07 18:05:22.942 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_20] [2022/07/07 18:05:22.942 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_20] [2022/07/07 18:05:23.023 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_20] ["msg id"=CKjjERDD+gEYACAA] [2022/07/07 18:05:23.023 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"=CKjjERDD+gEYACAA] [2022/07/07 18:05:23.028 +08:00] [INFO] [task.go:558] ["LoadSegmentTask Execute done"] [msgID=0] [2022/07/07 18:05:23.028 +08:00] [INFO] [impl.go:241] ["loadSegmentsTask WaitToFinish done"] [collectionID=434328479553028097] [segmentIDs="[434421958212583425]"] [nodeID=26] [2022/07/07 18:05:23.033 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:23.558 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=14] [2022/07/07 18:05:23.558 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=14] [addr=] [2022/07/07 18:05:23.558 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=14] [2022/07/07 18:05:23.558 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=14] [addr=] [2022/07/07 18:05:23.575 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"=null] [state=3] [2022/07/07 18:05:23.575 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_16_434328479540183041v4] [distribution=null] [2022/07/07 18:05:23.602 +08:00] [INFO] [impl.go:229] ["loadSegmentsTask Enqueue done"] [collectionID=434328479553028097] [segmentIDs="[434421957334138884]"] [nodeID=26] [2022/07/07 18:05:23.602 +08:00] [INFO] [task.go:507] ["LoadSegmentTask PreExecute start"] [msgID=0] [2022/07/07 18:05:23.603 +08:00] [INFO] [task.go:533] ["LoadSegmentTask PreExecute done"] [msgID=0] [2022/07/07 18:05:23.603 +08:00] [INFO] [task.go:539] ["LoadSegmentTask Execute start"] [msgID=0] [2022/07/07 18:05:23.603 +08:00] [INFO] [segment_loader.go:86] ["segmentLoader start loading..."] [collectionID=434328479553028097] [segmentNum=1] [segmentType=Sealed] [2022/07/07 18:05:23.603 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434405864160362497] [segmentID=434421957334138884] [segmentType=Sealed] [2022/07/07 18:05:23.603 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434405864160362497] [segmentID=434421957334138884] [segmentType=Sealed] [2022/07/07 18:05:23.778 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434421957334138884] [fieldID=102] [2022/07/07 18:05:23.778 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434421957334138884] [fieldID=102] [2022/07/07 18:05:23.791 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=33462] [segmentID=434421957334138884] [2022/07/07 18:05:23.792 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=33462] [segmentID=434421957334138884] [2022/07/07 18:05:23.793 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=33462] [segmentID=434421957334138884] [2022/07/07 18:05:23.809 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=33462] [segmentID=434421957334138884] [2022/07/07 18:05:23.809 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434421957334138884] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:23.960 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434421957334138884] [2022/07/07 18:05:23.988 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434405864160362497] [segmentID=434421957334138884] [segmentType=Sealed] [2022/07/07 18:05:23.988 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_23_434328479553028097v5" msgID:"\010\245\343\021\020\302\372\001\030\000 \000" msgGroup:"by-dev-dataNode-22-434328479553028097" timestamp:434421908693581827 "] ["msg id"=CKXjERDC+gEYACAA] [2022/07/07 18:05:23.993 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_23] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:23.999 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_23] [MessageID={}] [2022/07/07 18:05:24.116 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_23] [2022/07/07 18:05:24.116 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_23] [2022/07/07 18:05:24.175 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_23] ["msg id"=CKXjERDC+gEYACAA] [2022/07/07 18:05:24.175 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"=CKXjERDC+gEYACAA] [2022/07/07 18:05:24.177 +08:00] [INFO] [task.go:558] ["LoadSegmentTask Execute done"] [msgID=0] [2022/07/07 18:05:24.177 +08:00] [INFO] [impl.go:241] ["loadSegmentsTask WaitToFinish done"] [collectionID=434328479553028097] [segmentIDs="[434421957334138884]"] [nodeID=26] [2022/07/07 18:05:24.182 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:24.212 +08:00] [INFO] [impl.go:229] ["loadSegmentsTask Enqueue done"] [collectionID=434328479553028097] [segmentIDs="[434421957334138891]"] [nodeID=26] [2022/07/07 18:05:24.212 +08:00] [INFO] [task.go:507] ["LoadSegmentTask PreExecute start"] [msgID=0] [2022/07/07 18:05:24.212 +08:00] [INFO] [task.go:533] ["LoadSegmentTask PreExecute done"] [msgID=0] [2022/07/07 18:05:24.212 +08:00] [INFO] [task.go:539] ["LoadSegmentTask Execute start"] [msgID=0] [2022/07/07 18:05:24.212 +08:00] [INFO] [segment_loader.go:86] ["segmentLoader start loading..."] [collectionID=434328479553028097] [segmentNum=1] [segmentType=Sealed] [2022/07/07 18:05:24.212 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434405864160362497] [segmentID=434421957334138891] [segmentType=Sealed] [2022/07/07 18:05:24.212 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434405864160362497] [segmentID=434421957334138891] [segmentType=Sealed] [2022/07/07 18:05:24.396 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434421957334138891] [fieldID=102] [2022/07/07 18:05:24.396 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434421957334138891] [fieldID=102] [2022/07/07 18:05:24.407 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=33434] [segmentID=434421957334138891] [2022/07/07 18:05:24.424 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=33434] [segmentID=434421957334138891] [2022/07/07 18:05:24.424 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=33434] [segmentID=434421957334138891] [2022/07/07 18:05:24.424 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=33434] [segmentID=434421957334138891] [2022/07/07 18:05:24.424 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434421957334138891] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:24.571 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434421957334138891] [2022/07/07 18:05:24.602 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434405864160362497] [segmentID=434421957334138891] [segmentType=Sealed] [2022/07/07 18:05:24.602 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_18_434328479553028097v0" msgID:"\010\251\343\021\020\303\372\001\030\000 \000" msgGroup:"by-dev-dataNode-22-434328479553028097" timestamp:434421908746010629 "] ["msg id"=CKnjERDD+gEYACAA] [2022/07/07 18:05:24.607 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_18] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:24.616 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_18] [MessageID={}] [2022/07/07 18:05:24.723 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_18] [2022/07/07 18:05:24.723 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_18] [2022/07/07 18:05:24.795 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_18] ["msg id"=CKnjERDD+gEYACAA] [2022/07/07 18:05:24.795 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"=CKnjERDD+gEYACAA] [2022/07/07 18:05:24.796 +08:00] [INFO] [task.go:558] ["LoadSegmentTask Execute done"] [msgID=0] [2022/07/07 18:05:24.797 +08:00] [INFO] [impl.go:241] ["loadSegmentsTask WaitToFinish done"] [collectionID=434328479553028097] [segmentIDs="[434421957334138891]"] [nodeID=26] [2022/07/07 18:05:24.801 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:24.915 +08:00] [INFO] [impl.go:123] ["watchDmChannelsTask Enqueue done"] [collectionID=434328479448170498] [nodeID=26] [replicaID=434327753245589514] [2022/07/07 18:05:24.915 +08:00] [INFO] [task.go:146] ["Starting WatchDmChannels ..."] [collectionName=useralbumfraudmale] [collectionID=434328479448170498] [replicaID=434327753245589514] ["load type"=LoadCollection] [vChannels="[by-dev-rootcoord-dml_6_434328479448170498v0]"] [pChannels="[by-dev-rootcoord-dml_6]"] [2022/07/07 18:05:24.915 +08:00] [INFO] [collection.go:329] ["create collection"] [collectionID=434328479448170498] [2022/07/07 18:05:24.915 +08:00] [INFO] [shard_node_detector.go:68] ["nodeDetector watch"] [collectionID=434328479448170498] [replicaID=434327753245589514] [vchannelName=by-dev-rootcoord-dml_6_434328479448170498v0] [2022/07/07 18:05:24.917 +08:00] [INFO] [impl.go:229] ["loadSegmentsTask Enqueue done"] [collectionID=434328479553028097] [segmentIDs="[434359622417514498,434404055602692102,434404072615837697]"] [nodeID=26] [2022/07/07 18:05:24.917 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=9] [2022/07/07 18:05:24.917 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=12] [2022/07/07 18:05:24.917 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=10] [2022/07/07 18:05:24.918 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=26] [2022/07/07 18:05:24.918 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=29] [2022/07/07 18:05:24.918 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=28] [2022/07/07 18:05:24.918 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=32] [2022/07/07 18:05:24.918 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=27] [2022/07/07 18:05:24.918 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=31] [2022/07/07 18:05:24.918 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=30] [2022/07/07 18:05:24.918 +08:00] [INFO] [shard_segment_detector.go:74] ["segmentDetector start watch"] [collectionID=434328479448170498] [replicaID=434327753245589514] [vchannelName=by-dev-rootcoord-dml_6_434328479448170498v0] [rootPath=by-dev/meta/queryCoord-segmentMeta/434328479448170498] [2022/07/07 18:05:24.918 +08:00] [INFO] [shard_cluster_service.go:80] ["successfully add shard cluster"] [collectionID=434328479448170498] [replica=434327753245589514] [vchan=by-dev-rootcoord-dml_6_434328479448170498v0] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434328479448170499] [collectionID=434328479448170498] [partitions="[434328479448170499]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434328479448170499] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434328479906922497] [collectionID=434328479448170498] [partitions="[434328479448170499,434328479906922497]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434328479906922497] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434337916580003843] [collectionID=434328479448170498] [partitions="[434328479448170499,434328479906922497,434337916580003843]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434337916580003843] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434360565913092097] [collectionID=434328479448170498] [partitions="[434328479448170499,434328479906922497,434337916580003843,434360565913092097]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434360565913092097] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434383215128477697] [collectionID=434328479448170498] [partitions="[434328479448170499,434328479906922497,434337916580003843,434360565913092097,434383215128477697]"] [2022/07/07 18:05:24.919 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434383215128477697] [2022/07/07 18:05:24.919 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434405864160362499] [collectionID=434328479448170498] [partitions="[434328479448170499,434328479906922497,434337916580003843,434360565913092097,434383215128477697,434405864160362499]"] [2022/07/07 18:05:24.919 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434405864160362499] [2022/07/07 18:05:24.919 +08:00] [INFO] [task.go:218] ["loading growing segments in WatchDmChannels..."] [collectionID=434328479448170498] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:24.919 +08:00] [INFO] [task.go:227] ["successfully load growing segments done in WatchDmChannels"] [collectionID=434328479448170498] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:24.919 +08:00] [INFO] [task.go:259] ["watchDMChannel, group channels done"] [collectionID=434328479448170498] [2022/07/07 18:05:24.919 +08:00] [INFO] [task.go:274] ["watchDMChannel, add check points info for unflushed segments done"] [collectionID=434328479448170498] [unflushedSegmentIDs="[]"] [2022/07/07 18:05:24.919 +08:00] [INFO] [task.go:299] ["watchDMChannel, add check points info for flushed segments done"] [collectionID=434328479448170498] [flushedSegmentIDs="[]"] [2022/07/07 18:05:24.919 +08:00] [INFO] [task.go:324] ["watchDMChannel, add check points info for dropped segments done"] [collectionID=434328479448170498] [droppedSegmentIDs="[]"] [2022/07/07 18:05:24.919 +08:00] [INFO] [data_sync_service.go:99] ["add DML flow graph"] [collectionID=434328479448170498] [channel=by-dev-rootcoord-dml_6_434328479448170498v0] [2022/07/07 18:05:24.919 +08:00] [INFO] [task.go:335] ["Query node add DML flow graphs"] [collectionID=434328479448170498] [channels="[by-dev-rootcoord-dml_6_434328479448170498v0]"] [2022/07/07 18:05:24.924 +08:00] [INFO] [mq_msgstream.go:871] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_6] [MessageID={}] [2022/07/07 18:05:24.926 +08:00] [INFO] [impl.go:123] ["watchDmChannelsTask Enqueue done"] [collectionID=434328479553028097] [nodeID=26] [replicaID=434327753245589538] [2022/07/07 18:05:25.039 +08:00] [INFO] [mq_msgstream.go:877] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_6] [2022/07/07 18:05:25.053 +08:00] [INFO] [flow_graph_query_node.go:237] ["query node flow graph seeks from pChannel"] [collectionID=434328479448170498] [channel=by-dev-rootcoord-dml_6] [2022/07/07 18:05:25.053 +08:00] [INFO] [task.go:373] ["watchDMChannel, add flowGraph for dmChannels success"] [collectionID=434328479448170498] [vChannels="[by-dev-rootcoord-dml_6_434328479448170498v0]"] [2022/07/07 18:05:25.053 +08:00] [INFO] [collection.go:118] ["add vChannel to collection"] [collectionID=434328479448170498] [vChannel=by-dev-rootcoord-dml_6_434328479448170498v0] [2022/07/07 18:05:25.053 +08:00] [INFO] [collection.go:171] ["add pChannel to collection"] [collectionID=434328479448170498] [pChannel=by-dev-rootcoord-dml_6] [2022/07/07 18:05:25.053 +08:00] [INFO] [task.go:379] ["watchDMChannel, init replica done"] [collectionID=434328479448170498] [vChannels="[by-dev-rootcoord-dml_6_434328479448170498v0]"] [2022/07/07 18:05:25.053 +08:00] [INFO] [tsafe_replica.go:121] ["add tSafe done"] [channel=by-dev-rootcoord-dml_6_434328479448170498v0] [2022/07/07 18:05:25.053 +08:00] [INFO] [query_shard_service.go:96] ["Successfully add query shard"] [collection=434328479448170498] [replica=434327753245589514] [channel=by-dev-rootcoord-dml_6_434328479448170498v0] [2022/07/07 18:05:25.053 +08:00] [INFO] [task.go:398] ["WatchDmChannels done"] [collectionID=434328479448170498] [vChannels="[by-dev-rootcoord-dml_6_434328479448170498v0]"] [2022/07/07 18:05:25.053 +08:00] [INFO] [task.go:507] ["LoadSegmentTask PreExecute start"] [msgID=0] [2022/07/07 18:05:25.053 +08:00] [INFO] [task.go:533] ["LoadSegmentTask PreExecute done"] [msgID=0] [2022/07/07 18:05:25.053 +08:00] [INFO] [task.go:539] ["LoadSegmentTask Execute start"] [msgID=0] [2022/07/07 18:05:25.053 +08:00] [INFO] [segment_loader.go:86] ["segmentLoader start loading..."] [collectionID=434328479553028097] [segmentNum=3] [segmentType=Sealed] [2022/07/07 18:05:25.053 +08:00] [INFO] [impl.go:134] ["watchDmChannelsTask WaitToFinish done"] [collectionID=434328479448170498] [nodeID=26] [2022/07/07 18:05:25.053 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514498] [segmentType=Sealed] [2022/07/07 18:05:25.054 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404055602692102] [segmentType=Sealed] [2022/07/07 18:05:25.054 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404072615837697] [segmentType=Sealed] [2022/07/07 18:05:25.054 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404072615837697] [segmentType=Sealed] [2022/07/07 18:05:25.054 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404055602692102] [segmentType=Sealed] [2022/07/07 18:05:25.054 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514498] [segmentType=Sealed] [2022/07/07 18:05:25.205 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434359622417514498] [fieldID=102] [2022/07/07 18:05:25.205 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434359622417514498] [fieldID=102] [2022/07/07 18:05:25.211 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=4276] [segmentID=434359622417514498] [2022/07/07 18:05:25.211 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=4276] [segmentID=434359622417514498] [2022/07/07 18:05:25.212 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=4276] [segmentID=434359622417514498] [2022/07/07 18:05:25.212 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=4276] [segmentID=434359622417514498] [2022/07/07 18:05:25.212 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434359622417514498] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:25.240 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434359622417514498] [2022/07/07 18:05:25.365 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434404055602692102] [fieldID=102] [2022/07/07 18:05:25.365 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434404055602692102] [fieldID=102] [2022/07/07 18:05:25.381 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434404072615837697] [fieldID=102] [2022/07/07 18:05:25.381 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434404072615837697] [fieldID=102] [2022/07/07 18:05:25.383 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=48276] [segmentID=434404055602692102] [2022/07/07 18:05:25.399 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=51605] [segmentID=434404072615837697] [2022/07/07 18:05:25.414 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=48276] [segmentID=434404055602692102] [2022/07/07 18:05:25.414 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=48276] [segmentID=434404055602692102] [2022/07/07 18:05:25.415 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=48276] [segmentID=434404055602692102] [2022/07/07 18:05:25.415 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434404055602692102] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:25.427 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=51605] [segmentID=434404072615837697] [2022/07/07 18:05:25.427 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=51605] [segmentID=434404072615837697] [2022/07/07 18:05:25.429 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=51605] [segmentID=434404072615837697] [2022/07/07 18:05:25.429 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434404072615837697] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:25.578 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434404055602692102] [2022/07/07 18:05:25.649 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434404072615837697] [2022/07/07 18:05:25.658 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514498] [segmentType=Sealed] [2022/07/07 18:05:25.658 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404055602692102] [segmentType=Sealed] [2022/07/07 18:05:25.658 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404072615837697] [segmentType=Sealed] [2022/07/07 18:05:25.658 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_18_434328479553028097v0" msgID:"\010\274\224\021\020\000\030\000 \000" "] ["msg id"="CLyUERAAGAAgAA=="] [2022/07/07 18:05:25.664 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_18] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:25.673 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_18] [MessageID={}] [2022/07/07 18:05:25.778 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_18] [2022/07/07 18:05:25.778 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_18] [2022/07/07 18:05:27.677 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_18] ["msg id"="CLyUERAAGAAgAA=="] [2022/07/07 18:05:27.677 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CLyUERAAGAAgAA=="] [2022/07/07 18:05:27.683 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_22_434328479553028097v4" msgID:"\010\300\224\021\020\000\030\000 \000" "] ["msg id"="CMCUERAAGAAgAA=="] [2022/07/07 18:05:27.688 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_22] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:27.689 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_22] [MessageID={}] [2022/07/07 18:05:27.801 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_22] [2022/07/07 18:05:27.801 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_22] [2022/07/07 18:05:29.567 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_22] ["msg id"="CMCUERAAGAAgAA=="] [2022/07/07 18:05:29.567 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CMCUERAAGAAgAA=="] [2022/07/07 18:05:29.569 +08:00] [INFO] [task.go:558] ["LoadSegmentTask Execute done"] [msgID=0] [2022/07/07 18:05:29.569 +08:00] [INFO] [task.go:146] ["Starting WatchDmChannels ..."] [collectionName=useralbumfirstpicture] [collectionID=434328479553028097] [replicaID=434327753245589538] ["load type"=LoadCollection] [vChannels="[by-dev-rootcoord-dml_23_434328479553028097v5]"] [pChannels="[by-dev-rootcoord-dml_23]"] [2022/07/07 18:05:29.569 +08:00] [INFO] [shard_node_detector.go:68] ["nodeDetector watch"] [collectionID=434328479553028097] [replicaID=434327753245589538] [vchannelName=by-dev-rootcoord-dml_23_434328479553028097v5] [2022/07/07 18:05:29.569 +08:00] [INFO] [impl.go:241] ["loadSegmentsTask WaitToFinish done"] [collectionID=434328479553028097] [segmentIDs="[434359622417514498,434404055602692102,434404072615837697]"] [nodeID=26] [2022/07/07 18:05:29.571 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=12] [2022/07/07 18:05:29.571 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=9] [2022/07/07 18:05:29.571 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=10] [2022/07/07 18:05:29.572 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=26] [2022/07/07 18:05:29.572 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=30] [2022/07/07 18:05:29.572 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=27] [2022/07/07 18:05:29.572 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=31] [2022/07/07 18:05:29.572 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=29] [2022/07/07 18:05:29.572 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=28] [2022/07/07 18:05:29.572 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=32] [2022/07/07 18:05:29.572 +08:00] [INFO] [shard_segment_detector.go:74] ["segmentDetector start watch"] [collectionID=434328479553028097] [replicaID=434327753245589538] [vchannelName=by-dev-rootcoord-dml_23_434328479553028097v5] [rootPath=by-dev/meta/queryCoord-segmentMeta/434328479553028097] [2022/07/07 18:05:29.573 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434337948888989697] [state=3] [2022/07/07 18:05:29.573 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434358757119492106] [state=3] [2022/07/07 18:05:29.573 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434381423544631297] [state=3] [2022/07/07 18:05:29.573 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434382271632900098] [state=3] [2022/07/07 18:05:29.573 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434404071331332100] [state=3] [2022/07/07 18:05:29.573 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434404920612356097] [state=3] [2022/07/07 18:05:29.573 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434421957334138884] [state=3] [2022/07/07 18:05:29.573 +08:00] [INFO] [shard_cluster_service.go:80] ["successfully add shard cluster"] [collectionID=434328479553028097] [replica=434327753245589538] [vchan=by-dev-rootcoord-dml_23_434328479553028097v5] [2022/07/07 18:05:29.573 +08:00] [INFO] [task.go:218] ["loading growing segments in WatchDmChannels..."] [collectionID=434328479553028097] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:29.573 +08:00] [INFO] [task.go:227] ["successfully load growing segments done in WatchDmChannels"] [collectionID=434328479553028097] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:29.573 +08:00] [INFO] [task.go:259] ["watchDMChannel, group channels done"] [collectionID=434328479553028097] [2022/07/07 18:05:29.573 +08:00] [INFO] [task.go:274] ["watchDMChannel, add check points info for unflushed segments done"] [collectionID=434328479553028097] [unflushedSegmentIDs="[]"] [2022/07/07 18:05:29.573 +08:00] [INFO] [task.go:299] ["watchDMChannel, add check points info for flushed segments done"] [collectionID=434328479553028097] [flushedSegmentIDs="[434337948888989697,434358757119492106,434359622417514500,434381423544631297,434382271632900098,434404071331332100,434404920612356097,434421957334138884]"] [2022/07/07 18:05:29.573 +08:00] [INFO] [task.go:324] ["watchDMChannel, add check points info for dropped segments done"] [collectionID=434328479553028097] [droppedSegmentIDs="[434332285818372100,434329423363440642,434335132702474249,434330367016304641,434334141981392900,434332254505271297,434331310695383042,434336972953354244,434335085555613698,434333198223671298,434328480103530503,434336029221847041,434346410137354242,434351128965021697,434348376217092103,434352072539242500,434349241331613699,434354022798852100,434338860520964100,434344585614852103,434355847255818242,434343578929463297,434353959831863298,434340747840061441,434350247925514244,434350184919203842,434342635302813698,434344522543005698,434347353986826242,434358678489661441,434341691414020097,434353016153047043,434357734692618244,434348297665642497,434339804056125443,434354903943479298,434341723002634250,434337916763504643,434355910235652104,434356791052861441,434345466326941697,434371890796298241,434361509592170497,434370003228164100,434372834448900100,434370946697789444,434367219127812100,434363396911005699,434379440438378499,434378592035536897,434380384288112642,434372897166852108,434369059483749697,434381327822749697,434365284596842502,434367171797712901,434376609204797442,434364340904919042,434375744050954250,434374721780842497,434377553093328898,434364372532592641,434375665577885698,434366228118634499,434368115817578500,434373778049335300,434378496470941698,434360566162391041,434362453494071298,434371009730052100,434399258354384900,434402089654026243,434395577865732104,434383215351300099,434398314845700100,434397371022442499,434403977025028097,434388877320912899,434385102644183042,434386093771063297,434386046336106498,434400202217226242,434390764993904642,434394539841290241,434396427343364099,434387933838704641,434393596004663300,434398409020932105,434386990028554242,434392730982154248,434395483441663300,434392652417597444,434384158925783041,434389884098314244,434391708672983044,434401240176132105,434403033398640641,434401145869828097,434389821249290243,434411526588727298,434415301790007298,434420020080803842,434409639296106497,434407751951056897,434410582844112897,434419076598071298,434417189043306497,434409670989578242,434408695420420098,434405864435875843,434416245259370498,434414358071607297,434421907671220225,434415349264285697,434420964218634242,434418132840349698,434419139368452102,434406808232919044,434413414300778498,434412517978275841,434412470503735298]"] [2022/07/07 18:05:29.573 +08:00] [INFO] [data_sync_service.go:99] ["add DML flow graph"] [collectionID=434328479553028097] [channel=by-dev-rootcoord-dml_23_434328479553028097v5] [2022/07/07 18:05:29.573 +08:00] [INFO] [task.go:335] ["Query node add DML flow graphs"] [collectionID=434328479553028097] [channels="[by-dev-rootcoord-dml_23_434328479553028097v5]"] [2022/07/07 18:05:29.577 +08:00] [INFO] [mq_msgstream.go:871] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_23] [MessageID={}] [2022/07/07 18:05:29.693 +08:00] [INFO] [mq_msgstream.go:877] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_23] [2022/07/07 18:05:29.705 +08:00] [INFO] [flow_graph_query_node.go:237] ["query node flow graph seeks from pChannel"] [collectionID=434328479553028097] [channel=by-dev-rootcoord-dml_23] [2022/07/07 18:05:29.705 +08:00] [INFO] [task.go:373] ["watchDMChannel, add flowGraph for dmChannels success"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_23_434328479553028097v5]"] [2022/07/07 18:05:29.705 +08:00] [INFO] [collection.go:118] ["add vChannel to collection"] [collectionID=434328479553028097] [vChannel=by-dev-rootcoord-dml_23_434328479553028097v5] [2022/07/07 18:05:29.705 +08:00] [INFO] [collection.go:171] ["add pChannel to collection"] [collectionID=434328479553028097] [pChannel=by-dev-rootcoord-dml_23] [2022/07/07 18:05:29.705 +08:00] [INFO] [task.go:379] ["watchDMChannel, init replica done"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_23_434328479553028097v5]"] [2022/07/07 18:05:29.705 +08:00] [INFO] [tsafe_replica.go:121] ["add tSafe done"] [channel=by-dev-rootcoord-dml_23_434328479553028097v5] [2022/07/07 18:05:29.705 +08:00] [INFO] [task.go:398] ["WatchDmChannels done"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_23_434328479553028097v5]"] [2022/07/07 18:05:29.705 +08:00] [INFO] [impl.go:134] ["watchDmChannelsTask WaitToFinish done"] [collectionID=434328479553028097] [nodeID=26] [2022/07/07 18:05:29.709 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:29.714 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=12] [2022/07/07 18:05:29.714 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=12] [addr=] [2022/07/07 18:05:29.716 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=12] [2022/07/07 18:05:29.716 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=12] [addr=] [2022/07/07 18:05:29.716 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=12] [2022/07/07 18:05:29.717 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=12] [addr=] [2022/07/07 18:05:29.717 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=12] [2022/07/07 18:05:29.717 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=12] [addr=] [2022/07/07 18:05:29.937 +08:00] [INFO] [impl.go:229] ["loadSegmentsTask Enqueue done"] [collectionID=434328479553028101] [segmentIDs="[434359622234275842,434359622234275843,434359622247120897,434359622234275844,434359622234275841,434359622247120898]"] [nodeID=26] [2022/07/07 18:05:29.937 +08:00] [INFO] [task.go:507] ["LoadSegmentTask PreExecute start"] [msgID=0] [2022/07/07 18:05:29.937 +08:00] [INFO] [task.go:533] ["LoadSegmentTask PreExecute done"] [msgID=0] [2022/07/07 18:05:29.937 +08:00] [INFO] [task.go:539] ["LoadSegmentTask Execute start"] [msgID=0] [2022/07/07 18:05:29.937 +08:00] [INFO] [segment_loader.go:86] ["segmentLoader start loading..."] [collectionID=434328479553028101] [segmentNum=6] [segmentType=Sealed] [2022/07/07 18:05:29.937 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275842] [segmentType=Sealed] [2022/07/07 18:05:29.937 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275843] [segmentType=Sealed] [2022/07/07 18:05:29.937 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622247120897] [segmentType=Sealed] [2022/07/07 18:05:29.937 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275844] [segmentType=Sealed] [2022/07/07 18:05:29.937 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275841] [segmentType=Sealed] [2022/07/07 18:05:29.937 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622247120898] [segmentType=Sealed] [2022/07/07 18:05:29.938 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622247120898] [segmentType=Sealed] [2022/07/07 18:05:29.938 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275842] [segmentType=Sealed] [2022/07/07 18:05:29.938 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622247120897] [segmentType=Sealed] [2022/07/07 18:05:29.938 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275843] [segmentType=Sealed] [2022/07/07 18:05:29.938 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275844] [segmentType=Sealed] [2022/07/07 18:05:29.938 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275841] [segmentType=Sealed] [2022/07/07 18:05:29.946 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=843] [segmentID=434359622247120897] [2022/07/07 18:05:29.946 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=838] [segmentID=434359622234275842] [2022/07/07 18:05:29.946 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=838] [segmentID=434359622234275842] [2022/07/07 18:05:29.946 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=838] [segmentID=434359622234275842] [2022/07/07 18:05:29.946 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=859] [segmentID=434359622247120898] [2022/07/07 18:05:29.947 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=848] [segmentID=434359622234275843] [2022/07/07 18:05:29.947 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=890] [segmentID=434359622234275844] [2022/07/07 18:05:29.947 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=848] [segmentID=434359622234275843] [2022/07/07 18:05:29.947 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=848] [segmentID=434359622234275843] [2022/07/07 18:05:29.947 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=859] [segmentID=434359622247120898] [2022/07/07 18:05:29.947 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=838] [segmentID=434359622234275842] [2022/07/07 18:05:29.948 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=890] [segmentID=434359622234275844] [2022/07/07 18:05:29.948 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=842] [segmentID=434359622234275841] [2022/07/07 18:05:29.948 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=859] [segmentID=434359622247120898] [2022/07/07 18:05:29.948 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=859] [segmentID=434359622247120898] [2022/07/07 18:05:29.948 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=848] [segmentID=434359622234275843] [2022/07/07 18:05:29.948 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=842] [segmentID=434359622234275841] [2022/07/07 18:05:29.948 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=843] [segmentID=434359622247120897] [2022/07/07 18:05:29.949 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=890] [segmentID=434359622234275844] [2022/07/07 18:05:29.949 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=890] [segmentID=434359622234275844] [2022/07/07 18:05:29.949 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=843] [segmentID=434359622247120897] [2022/07/07 18:05:29.949 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=842] [segmentID=434359622234275841] [2022/07/07 18:05:29.949 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=843] [segmentID=434359622247120897] [2022/07/07 18:05:29.949 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=842] [segmentID=434359622234275841] [2022/07/07 18:05:29.982 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=102] ["row count"=838] [segmentID=434359622234275842] [2022/07/07 18:05:29.983 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028101] [segment=434359622234275842] [len(field)=5] [segmentType=Sealed] [2022/07/07 18:05:29.985 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=102] ["row count"=859] [segmentID=434359622247120898] [2022/07/07 18:05:29.985 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028101] [segment=434359622247120898] [len(field)=5] [segmentType=Sealed] [2022/07/07 18:05:29.988 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=102] ["row count"=890] [segmentID=434359622234275844] [2022/07/07 18:05:29.988 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028101] [segment=434359622234275844] [len(field)=5] [segmentType=Sealed] [2022/07/07 18:05:29.992 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=102] ["row count"=848] [segmentID=434359622234275843] [2022/07/07 18:05:29.992 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028101] [segment=434359622234275843] [len(field)=5] [segmentType=Sealed] [2022/07/07 18:05:29.993 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=102] ["row count"=843] [segmentID=434359622247120897] [2022/07/07 18:05:29.993 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028101] [segment=434359622247120897] [len(field)=5] [segmentType=Sealed] [2022/07/07 18:05:29.998 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=102] ["row count"=842] [segmentID=434359622234275841] [2022/07/07 18:05:29.998 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028101] [segment=434359622234275841] [len(field)=5] [segmentType=Sealed] [2022/07/07 18:05:29.999 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434359622234275842] [2022/07/07 18:05:30.000 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434359622247120898] [2022/07/07 18:05:30.026 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434359622234275844] [2022/07/07 18:05:30.028 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434359622234275843] [2022/07/07 18:05:30.032 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434359622247120897] [2022/07/07 18:05:30.033 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434359622234275841] [2022/07/07 18:05:30.040 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622247120897] [segmentType=Sealed] [2022/07/07 18:05:30.040 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275844] [segmentType=Sealed] [2022/07/07 18:05:30.040 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275841] [segmentType=Sealed] [2022/07/07 18:05:30.040 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622247120898] [segmentType=Sealed] [2022/07/07 18:05:30.040 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275842] [segmentType=Sealed] [2022/07/07 18:05:30.040 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434337916580003841] [segmentID=434359622234275843] [segmentType=Sealed] [2022/07/07 18:05:30.040 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_31_434328479553028101v1" msgID:"\010\311\224\021\020\000\030\000 \000" "] ["msg id"="CMmUERAAGAAgAA=="] [2022/07/07 18:05:30.046 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_31] [subname=querynode-26-434328479553028101] [2022/07/07 18:05:30.047 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_31] [MessageID={}] [2022/07/07 18:05:30.167 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_31] [2022/07/07 18:05:30.167 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_31] [2022/07/07 18:05:32.006 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_31] ["msg id"="CMmUERAAGAAgAA=="] [2022/07/07 18:05:32.006 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CMmUERAAGAAgAA=="] [2022/07/07 18:05:32.008 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_32_434328479553028101v2" msgID:"\010\312\224\021\020\000\030\000 \000" "] ["msg id"="CMqUERAAGAAgAA=="] [2022/07/07 18:05:32.012 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_32] [subname=querynode-26-434328479553028101] [2022/07/07 18:05:32.021 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_32] [MessageID={}] [2022/07/07 18:05:32.124 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_32] [2022/07/07 18:05:32.124 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_32] [2022/07/07 18:05:33.689 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_32] ["msg id"="CMqUERAAGAAgAA=="] [2022/07/07 18:05:33.689 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CMqUERAAGAAgAA=="] [2022/07/07 18:05:33.691 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_34_434328479553028101v4" msgID:"\010\314\224\021\020\000\030\000 \000" "] ["msg id"="CMyUERAAGAAgAA=="] [2022/07/07 18:05:33.696 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_34] [subname=querynode-26-434328479553028101] [2022/07/07 18:05:33.704 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_34] [MessageID={}] [2022/07/07 18:05:33.828 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_34] [2022/07/07 18:05:33.828 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_34] [2022/07/07 18:05:36.030 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_34] ["msg id"="CMyUERAAGAAgAA=="] [2022/07/07 18:05:36.031 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CMyUERAAGAAgAA=="] [2022/07/07 18:05:36.038 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_33_434328479553028101v3" msgID:"\010\313\224\021\020\000\030\000 \000" "] ["msg id"="CMuUERAAGAAgAA=="] [2022/07/07 18:05:36.042 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_33] [subname=querynode-26-434328479553028101] [2022/07/07 18:05:36.048 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_33] [MessageID={}] [2022/07/07 18:05:36.156 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_33] [2022/07/07 18:05:36.156 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_33] [2022/07/07 18:05:38.403 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_33] ["msg id"="CMuUERAAGAAgAA=="] [2022/07/07 18:05:38.403 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CMuUERAAGAAgAA=="] [2022/07/07 18:05:38.412 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_35_434328479553028101v5" msgID:"\010\315\224\021\020\000\030\000 \000" "] ["msg id"="CM2UERAAGAAgAA=="] [2022/07/07 18:05:38.416 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_35] [subname=querynode-26-434328479553028101] [2022/07/07 18:05:38.417 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_35] [MessageID={}] [2022/07/07 18:05:38.539 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_35] [2022/07/07 18:05:38.539 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_35] [2022/07/07 18:05:40.032 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_35] ["msg id"="CM2UERAAGAAgAA=="] [2022/07/07 18:05:40.032 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CM2UERAAGAAgAA=="] [2022/07/07 18:05:40.034 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_30_434328479553028101v0" msgID:"\010\310\224\021\020\000\030\000 \000" "] ["msg id"="CMiUERAAGAAgAA=="] [2022/07/07 18:05:40.038 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_30] [subname=querynode-26-434328479553028101] [2022/07/07 18:05:40.049 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_30] [MessageID={}] [2022/07/07 18:05:40.170 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_30] [2022/07/07 18:05:40.170 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_30] [2022/07/07 18:05:42.493 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028101] [channel=by-dev-rootcoord-dml_30] ["msg id"="CMiUERAAGAAgAA=="] [2022/07/07 18:05:42.493 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CMiUERAAGAAgAA=="] [2022/07/07 18:05:42.502 +08:00] [INFO] [task.go:558] ["LoadSegmentTask Execute done"] [msgID=0] [2022/07/07 18:05:42.502 +08:00] [INFO] [impl.go:241] ["loadSegmentsTask WaitToFinish done"] [collectionID=434328479553028101] [segmentIDs="[434359622234275842,434359622234275843,434359622247120897,434359622234275844,434359622234275841,434359622247120898]"] [nodeID=26] [2022/07/07 18:05:42.507 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:42.512 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.512 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.514 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.514 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.515 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.515 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.515 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.515 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.892 +08:00] [INFO] [impl.go:229] ["loadSegmentsTask Enqueue done"] [collectionID=434328479553028097] [segmentIDs="[434337948128772102,434337948128772106,434337948128772104,434359622417514499,434359622417514500,434359622417514497,434381422089732104,434381422089732100,434381422089732102,434404071331332106]"] [nodeID=26] [2022/07/07 18:05:42.892 +08:00] [INFO] [task.go:507] ["LoadSegmentTask PreExecute start"] [msgID=0] [2022/07/07 18:05:42.892 +08:00] [INFO] [task.go:533] ["LoadSegmentTask PreExecute done"] [msgID=0] [2022/07/07 18:05:42.892 +08:00] [INFO] [task.go:539] ["LoadSegmentTask Execute start"] [msgID=0] [2022/07/07 18:05:42.892 +08:00] [INFO] [segment_loader.go:86] ["segmentLoader start loading..."] [collectionID=434328479553028097] [segmentNum=10] [segmentType=Sealed] [2022/07/07 18:05:42.898 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434328479906922500] [segmentID=434337948128772102] [segmentType=Sealed] [2022/07/07 18:05:42.898 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434328479906922500] [segmentID=434337948128772106] [segmentType=Sealed] [2022/07/07 18:05:42.898 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434328479906922500] [segmentID=434337948128772104] [segmentType=Sealed] [2022/07/07 18:05:42.898 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514499] [segmentType=Sealed] [2022/07/07 18:05:42.898 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514500] [segmentType=Sealed] [2022/07/07 18:05:42.900 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514497] [segmentType=Sealed] [2022/07/07 18:05:42.902 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434360565899984899] [segmentID=434381422089732104] [segmentType=Sealed] [2022/07/07 18:05:42.903 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434360565899984899] [segmentID=434381422089732100] [segmentType=Sealed] [2022/07/07 18:05:42.905 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434360565899984899] [segmentID=434381422089732102] [segmentType=Sealed] [2022/07/07 18:05:42.907 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404071331332106] [segmentType=Sealed] [2022/07/07 18:05:42.907 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404071331332106] [segmentType=Sealed] [2022/07/07 18:05:42.925 +08:00] [INFO] [impl.go:123] ["watchDmChannelsTask Enqueue done"] [collectionID=434328479553028097] [nodeID=26] [replicaID=434327753245589538] [2022/07/07 18:05:42.926 +08:00] [INFO] [impl.go:123] ["watchDmChannelsTask Enqueue done"] [collectionID=434328479553028099] [nodeID=26] [replicaID=434327753245589530] [2022/07/07 18:05:42.926 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434328479906922500] [segmentID=434337948128772102] [segmentType=Sealed] [2022/07/07 18:05:42.927 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434328479906922500] [segmentID=434337948128772106] [segmentType=Sealed] [2022/07/07 18:05:42.929 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434328479906922500] [segmentID=434337948128772104] [segmentType=Sealed] [2022/07/07 18:05:42.931 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514499] [segmentType=Sealed] [2022/07/07 18:05:42.934 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514500] [segmentType=Sealed] [2022/07/07 18:05:42.938 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514497] [segmentType=Sealed] [2022/07/07 18:05:42.940 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434360565899984899] [segmentID=434381422089732104] [segmentType=Sealed] [2022/07/07 18:05:42.942 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434360565899984899] [segmentID=434381422089732100] [segmentType=Sealed] [2022/07/07 18:05:42.942 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434360565899984899] [segmentID=434381422089732102] [segmentType=Sealed] [2022/07/07 18:05:43.371 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434359622417514499] [fieldID=102] [2022/07/07 18:05:43.371 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434359622417514499] [fieldID=102] [2022/07/07 18:05:43.635 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=4346] [segmentID=434359622417514499] [2022/07/07 18:05:43.704 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=4346] [segmentID=434359622417514499] [2022/07/07 18:05:43.726 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434359622417514497] [fieldID=102] [2022/07/07 18:05:43.726 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434359622417514497] [fieldID=102] [2022/07/07 18:05:43.759 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=4346] [segmentID=434359622417514499] [2022/07/07 18:05:43.768 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=4375] [segmentID=434359622417514497] [2022/07/07 18:05:43.770 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=4346] [segmentID=434359622417514499] [2022/07/07 18:05:43.770 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434359622417514499] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:43.792 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=4375] [segmentID=434359622417514497] [2022/07/07 18:05:43.794 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=4375] [segmentID=434359622417514497] [2022/07/07 18:05:43.797 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434359622417514500] [fieldID=102] [2022/07/07 18:05:43.798 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434359622417514500] [fieldID=102] [2022/07/07 18:05:43.799 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=4375] [segmentID=434359622417514497] [2022/07/07 18:05:43.799 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434359622417514497] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:43.805 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=4321] [segmentID=434359622417514500] [2022/07/07 18:05:43.805 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=4321] [segmentID=434359622417514500] [2022/07/07 18:05:43.825 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=4321] [segmentID=434359622417514500] [2022/07/07 18:05:43.831 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=4321] [segmentID=434359622417514500] [2022/07/07 18:05:43.831 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434359622417514500] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:43.833 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434359622417514497] [2022/07/07 18:05:43.834 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434359622417514499] [2022/07/07 18:05:43.852 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434359622417514500] [2022/07/07 18:05:44.058 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434337948128772106] [fieldID=102] [2022/07/07 18:05:44.058 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434337948128772106] [fieldID=102] [2022/07/07 18:05:44.136 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=26810] [segmentID=434337948128772106] [2022/07/07 18:05:44.138 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=26810] [segmentID=434337948128772106] [2022/07/07 18:05:44.164 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=26810] [segmentID=434337948128772106] [2022/07/07 18:05:44.164 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=26810] [segmentID=434337948128772106] [2022/07/07 18:05:44.164 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434337948128772106] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:44.188 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434337948128772104] [fieldID=102] [2022/07/07 18:05:44.188 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434337948128772104] [fieldID=102] [2022/07/07 18:05:44.198 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=26608] [segmentID=434337948128772104] [2022/07/07 18:05:44.201 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434337948128772102] [fieldID=102] [2022/07/07 18:05:44.201 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434337948128772102] [fieldID=102] [2022/07/07 18:05:44.214 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=26732] [segmentID=434337948128772102] [2022/07/07 18:05:44.216 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=26732] [segmentID=434337948128772102] [2022/07/07 18:05:44.217 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=26732] [segmentID=434337948128772102] [2022/07/07 18:05:44.220 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=26608] [segmentID=434337948128772104] [2022/07/07 18:05:44.220 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=26608] [segmentID=434337948128772104] [2022/07/07 18:05:44.220 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=26608] [segmentID=434337948128772104] [2022/07/07 18:05:44.220 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434337948128772104] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:44.232 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434404071331332106] [fieldID=102] [2022/07/07 18:05:44.233 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434404071331332106] [fieldID=102] [2022/07/07 18:05:44.241 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=26732] [segmentID=434337948128772102] [2022/07/07 18:05:44.242 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434337948128772102] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:44.249 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=51976] [segmentID=434404071331332106] [2022/07/07 18:05:44.257 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434381422089732104] [fieldID=102] [2022/07/07 18:05:44.257 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434381422089732104] [fieldID=102] [2022/07/07 18:05:44.273 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=53520] [segmentID=434381422089732104] [2022/07/07 18:05:44.298 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434337948128772106] [2022/07/07 18:05:44.298 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=51976] [segmentID=434404071331332106] [2022/07/07 18:05:44.299 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=51976] [segmentID=434404071331332106] [2022/07/07 18:05:44.300 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=51976] [segmentID=434404071331332106] [2022/07/07 18:05:44.300 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434404071331332106] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:44.315 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434381422089732102] [fieldID=102] [2022/07/07 18:05:44.315 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434381422089732102] [fieldID=102] [2022/07/07 18:05:44.438 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434337948128772104] [2022/07/07 18:05:44.444 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=53520] [segmentID=434381422089732104] [2022/07/07 18:05:44.445 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=53520] [segmentID=434381422089732104] [2022/07/07 18:05:44.445 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=53520] [segmentID=434381422089732104] [2022/07/07 18:05:44.445 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434381422089732104] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:44.448 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=53330] [segmentID=434381422089732102] [2022/07/07 18:05:44.451 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434381422089732100] [fieldID=102] [2022/07/07 18:05:44.451 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434381422089732100] [fieldID=102] [2022/07/07 18:05:44.458 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434337948128772102] [2022/07/07 18:05:44.459 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=53330] [segmentID=434381422089732102] [2022/07/07 18:05:44.460 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=53330] [segmentID=434381422089732102] [2022/07/07 18:05:44.469 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=53405] [segmentID=434381422089732100] [2022/07/07 18:05:44.494 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=53330] [segmentID=434381422089732102] [2022/07/07 18:05:44.495 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434381422089732102] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:44.503 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=53405] [segmentID=434381422089732100] [2022/07/07 18:05:44.503 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=53405] [segmentID=434381422089732100] [2022/07/07 18:05:44.503 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=53405] [segmentID=434381422089732100] [2022/07/07 18:05:44.503 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434381422089732100] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:44.644 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434404071331332106] [2022/07/07 18:05:44.662 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434381422089732104] [2022/07/07 18:05:44.716 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434381422089732102] [2022/07/07 18:05:44.725 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434381422089732100] [2022/07/07 18:05:44.732 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434360565899984899] [segmentID=434381422089732104] [segmentType=Sealed] [2022/07/07 18:05:44.732 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434328479906922500] [segmentID=434337948128772104] [segmentType=Sealed] [2022/07/07 18:05:44.732 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514497] [segmentType=Sealed] [2022/07/07 18:05:44.732 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514499] [segmentType=Sealed] [2022/07/07 18:05:44.732 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434337916592848898] [segmentID=434359622417514500] [segmentType=Sealed] [2022/07/07 18:05:44.732 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434360565899984899] [segmentID=434381422089732100] [segmentType=Sealed] [2022/07/07 18:05:44.732 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434360565899984899] [segmentID=434381422089732102] [segmentType=Sealed] [2022/07/07 18:05:44.732 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404071331332106] [segmentType=Sealed] [2022/07/07 18:05:44.732 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434328479906922500] [segmentID=434337948128772102] [segmentType=Sealed] [2022/07/07 18:05:44.732 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434328479906922500] [segmentID=434337948128772106] [segmentType=Sealed] [2022/07/07 18:05:44.732 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_20_434328479553028097v2" msgID:"\010\276\224\021\020\000\030\000 \000" "] ["msg id"="CL6UERAAGAAgAA=="] [2022/07/07 18:05:44.746 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_20] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:44.758 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_20] [MessageID={}] [2022/07/07 18:05:44.874 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_20] [2022/07/07 18:05:44.874 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_20] [2022/07/07 18:05:47.658 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_20] ["msg id"="CL6UERAAGAAgAA=="] [2022/07/07 18:05:47.658 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CL6UERAAGAAgAA=="] [2022/07/07 18:05:47.663 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_19_434328479553028097v1" msgID:"\010\275\224\021\020\000\030\000 \000" "] ["msg id"="CL2UERAAGAAgAA=="] [2022/07/07 18:05:47.668 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_19] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:47.676 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_19] [MessageID={}] [2022/07/07 18:05:47.794 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_19] [2022/07/07 18:05:47.794 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_19] [2022/07/07 18:05:50.253 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_19] ["msg id"="CL2UERAAGAAgAA=="] [2022/07/07 18:05:50.253 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CL2UERAAGAAgAA=="] [2022/07/07 18:05:50.258 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_18_434328479553028097v0" msgID:"\010\274\224\021\020\000\030\000 \000" "] ["msg id"="CLyUERAAGAAgAA=="] [2022/07/07 18:05:50.263 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_18] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:50.270 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_18] [MessageID={}] [2022/07/07 18:05:50.382 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_18] [2022/07/07 18:05:50.382 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_18] [2022/07/07 18:05:52.113 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_18] ["msg id"="CLyUERAAGAAgAA=="] [2022/07/07 18:05:52.113 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CLyUERAAGAAgAA=="] [2022/07/07 18:05:52.115 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_23_434328479553028097v5" msgID:"\010\301\224\021\020\000\030\000 \000" "] ["msg id"="CMGUERAAGAAgAA=="] [2022/07/07 18:05:52.118 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_23] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:52.127 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_23] [MessageID={}] [2022/07/07 18:05:52.231 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_23] [2022/07/07 18:05:52.231 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_23] [2022/07/07 18:05:53.874 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_23] ["msg id"="CMGUERAAGAAgAA=="] [2022/07/07 18:05:53.874 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CMGUERAAGAAgAA=="] [2022/07/07 18:05:53.876 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_21_434328479553028097v3" msgID:"\010\277\224\021\020\000\030\000 \000" "] ["msg id"="CL+UERAAGAAgAA=="] [2022/07/07 18:05:53.880 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_21] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:53.888 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_21] [MessageID={}] [2022/07/07 18:05:54.007 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_21] [2022/07/07 18:05:54.007 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_21] [2022/07/07 18:05:56.541 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_21] ["msg id"="CL+UERAAGAAgAA=="] [2022/07/07 18:05:56.541 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CL+UERAAGAAgAA=="] [2022/07/07 18:05:56.549 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_22_434328479553028097v4" msgID:"\010\300\224\021\020\000\030\000 \000" "] ["msg id"="CMCUERAAGAAgAA=="] [2022/07/07 18:05:56.554 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_22] [subname=querynode-26-434328479553028097] [2022/07/07 18:05:56.562 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_22] [MessageID={}] [2022/07/07 18:05:56.667 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_22] [2022/07/07 18:05:56.667 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_22] [2022/07/07 18:05:58.438 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_22] ["msg id"="CMCUERAAGAAgAA=="] [2022/07/07 18:05:58.438 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CMCUERAAGAAgAA=="] [2022/07/07 18:05:58.440 +08:00] [INFO] [task.go:558] ["LoadSegmentTask Execute done"] [msgID=0] [2022/07/07 18:05:58.440 +08:00] [INFO] [task.go:146] ["Starting WatchDmChannels ..."] [collectionName=useralbumfirstpicture] [collectionID=434328479553028097] [replicaID=434327753245589538] ["load type"=LoadCollection] [vChannels="[by-dev-rootcoord-dml_18_434328479553028097v0]"] [pChannels="[by-dev-rootcoord-dml_18]"] [2022/07/07 18:05:58.440 +08:00] [INFO] [shard_node_detector.go:68] ["nodeDetector watch"] [collectionID=434328479553028097] [replicaID=434327753245589538] [vchannelName=by-dev-rootcoord-dml_18_434328479553028097v0] [2022/07/07 18:05:58.440 +08:00] [INFO] [impl.go:241] ["loadSegmentsTask WaitToFinish done"] [collectionID=434328479553028097] [segmentIDs="[434337948128772102,434337948128772106,434337948128772104,434359622417514499,434359622417514500,434359622417514497,434381422089732104,434381422089732100,434381422089732102,434404071331332106]"] [nodeID=26] [2022/07/07 18:05:58.443 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=10] [2022/07/07 18:05:58.443 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=26] [2022/07/07 18:05:58.443 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=30] [2022/07/07 18:05:58.443 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=31] [2022/07/07 18:05:58.443 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=29] [2022/07/07 18:05:58.443 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=27] [2022/07/07 18:05:58.443 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=28] [2022/07/07 18:05:58.443 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=32] [2022/07/07 18:05:58.443 +08:00] [INFO] [shard_segment_detector.go:74] ["segmentDetector start watch"] [collectionID=434328479553028097] [replicaID=434327753245589538] [vchannelName=by-dev-rootcoord-dml_18_434328479553028097v0] [rootPath=by-dev/meta/queryCoord-segmentMeta/434328479553028097] [2022/07/07 18:05:58.444 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434358758548176897] [state=3] [2022/07/07 18:05:58.444 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434359622417514498] [state=3] [2022/07/07 18:05:58.444 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434382271619792898] [state=3] [2022/07/07 18:05:58.444 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434403977025028098] [state=3] [2022/07/07 18:05:58.444 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434404055602692102] [state=3] [2022/07/07 18:05:58.444 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434404920599248898] [state=3] [2022/07/07 18:05:58.444 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434421957334138891] [state=3] [2022/07/07 18:05:58.444 +08:00] [INFO] [shard_cluster_service.go:80] ["successfully add shard cluster"] [collectionID=434328479553028097] [replica=434327753245589538] [vchan=by-dev-rootcoord-dml_18_434328479553028097v0] [2022/07/07 18:05:58.444 +08:00] [INFO] [task.go:218] ["loading growing segments in WatchDmChannels..."] [collectionID=434328479553028097] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:58.444 +08:00] [INFO] [task.go:227] ["successfully load growing segments done in WatchDmChannels"] [collectionID=434328479553028097] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:58.444 +08:00] [INFO] [task.go:259] ["watchDMChannel, group channels done"] [collectionID=434328479553028097] [2022/07/07 18:05:58.444 +08:00] [INFO] [task.go:274] ["watchDMChannel, add check points info for unflushed segments done"] [collectionID=434328479553028097] [unflushedSegmentIDs="[]"] [2022/07/07 18:05:58.444 +08:00] [INFO] [task.go:299] ["watchDMChannel, add check points info for flushed segments done"] [collectionID=434328479553028097] [flushedSegmentIDs="[434337948128772104,434359622417514498,434358758548176897,434382271619792898,434381422089732104,434404920599248898,434403977025028098,434404055602692102,434421957334138891]"] [2022/07/07 18:05:58.445 +08:00] [INFO] [task.go:324] ["watchDMChannel, add check points info for dropped segments done"] [collectionID=434328479553028097] [droppedSegmentIDs="[434332254505271298,434330367002935298,434336972966199298,434331310682013698,434329423363440641,434335133252714497,434332285818372106,434336029234692100,434334141981392898,434335085542506499,434328480103530502,434333198223671300,434346410123984898,434347353986826241,434350184905834500,434338860508119041,434358678502768641,434344569886212102,434344522529898499,434349241331613698,434337916763504645,434346457323274245,434353959818756097,434356791039692098,434352072552611841,434350247925514246,434340747853168641,434354903956848644,434342635315920898,434341723002634246,434355847268925442,434341691427127299,434357734692618241,434353016139939842,434345466314096642,434351128951914497,434343578916356098,434339804056125444,434354038527754246,434355910235652100,434348297652797441,434372834448900097,434363396898160641,434371890809405442,434367234856452101,434371010293661697,434362453507178499,434364372244234246,434374721780842498,434364340904919045,434367171797712899,434360566175498241,434380384288112641,434366228118634500,434373778049335299,434369059496656899,434378590934532104,434377553106698241,434375744050954248,434372897166852102,434361509592170500,434368115817578498,434365284596842497,434370946697789443,434379440451485697,434375665577885697,434378496484048899,434376609204797444,434381327822749701,434370003228164098,434398314845700098,434384158912413698,434394539827920899,434389868369412100,434399258341539842,434402089654026244,434393596004663298,434400202217226241,434386093495812100,434385102617706497,434392715253252102,434403033398640643,434386990015184900,434392652404690241,434386046349213699,434387933851549697,434401145883197442,434391708659613698,434397371022442498,434389821249290244,434388877320912900,434396427343364097,434383215351300098,434395562137092106,434398393633079297,434401224447492102,434395483454832644,434390764993904646,434419076598071297,434415349054570497,434406808219811842,434407751964164098,434409670727172100,434414358071607298,434420964218634241,434415301803114498,434418132853456898,434417189056413700,434419139997597697,434421907671220226,434413414300778500,434408695420420097,434412517611012104,434410582831005697,434411526576144385,434405864435875842,434420020067434498,434409639282999297,434412470490628097,434416245259370497]"] [2022/07/07 18:05:58.445 +08:00] [INFO] [data_sync_service.go:99] ["add DML flow graph"] [collectionID=434328479553028097] [channel=by-dev-rootcoord-dml_18_434328479553028097v0] [2022/07/07 18:05:58.445 +08:00] [INFO] [task.go:335] ["Query node add DML flow graphs"] [collectionID=434328479553028097] [channels="[by-dev-rootcoord-dml_18_434328479553028097v0]"] [2022/07/07 18:05:58.449 +08:00] [INFO] [mq_msgstream.go:871] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_18] [MessageID={}] [2022/07/07 18:05:58.555 +08:00] [INFO] [mq_msgstream.go:877] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_18] [2022/07/07 18:05:58.565 +08:00] [INFO] [flow_graph_query_node.go:237] ["query node flow graph seeks from pChannel"] [collectionID=434328479553028097] [channel=by-dev-rootcoord-dml_18] [2022/07/07 18:05:58.565 +08:00] [INFO] [task.go:373] ["watchDMChannel, add flowGraph for dmChannels success"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_18_434328479553028097v0]"] [2022/07/07 18:05:58.565 +08:00] [INFO] [collection.go:118] ["add vChannel to collection"] [collectionID=434328479553028097] [vChannel=by-dev-rootcoord-dml_18_434328479553028097v0] [2022/07/07 18:05:58.565 +08:00] [INFO] [collection.go:171] ["add pChannel to collection"] [collectionID=434328479553028097] [pChannel=by-dev-rootcoord-dml_18] [2022/07/07 18:05:58.565 +08:00] [INFO] [task.go:379] ["watchDMChannel, init replica done"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_18_434328479553028097v0]"] [2022/07/07 18:05:58.565 +08:00] [INFO] [tsafe_replica.go:121] ["add tSafe done"] [channel=by-dev-rootcoord-dml_18_434328479553028097v0] [2022/07/07 18:05:58.565 +08:00] [INFO] [task.go:398] ["WatchDmChannels done"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_18_434328479553028097v0]"] [2022/07/07 18:05:58.565 +08:00] [INFO] [task.go:146] ["Starting WatchDmChannels ..."] [collectionName=useralbumpolitical] [collectionID=434328479553028099] [replicaID=434327753245589530] ["load type"=LoadCollection] [vChannels="[by-dev-rootcoord-dml_29_434328479553028099v5]"] [pChannels="[by-dev-rootcoord-dml_29]"] [2022/07/07 18:05:58.565 +08:00] [INFO] [impl.go:134] ["watchDmChannelsTask WaitToFinish done"] [collectionID=434328479553028097] [nodeID=26] [2022/07/07 18:05:58.565 +08:00] [INFO] [collection.go:329] ["create collection"] [collectionID=434328479553028099] [2022/07/07 18:05:58.565 +08:00] [INFO] [shard_node_detector.go:68] ["nodeDetector watch"] [collectionID=434328479553028099] [replicaID=434327753245589530] [vchannelName=by-dev-rootcoord-dml_29_434328479553028099v5] [2022/07/07 18:05:58.567 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=10] [2022/07/07 18:05:58.567 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=26] [2022/07/07 18:05:58.567 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=31] [2022/07/07 18:05:58.567 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=30] [2022/07/07 18:05:58.567 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=32] [2022/07/07 18:05:58.567 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=27] [2022/07/07 18:05:58.567 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=29] [2022/07/07 18:05:58.567 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=28] [2022/07/07 18:05:58.567 +08:00] [INFO] [shard_segment_detector.go:74] ["segmentDetector start watch"] [collectionID=434328479553028099] [replicaID=434327753245589530] [vchannelName=by-dev-rootcoord-dml_29_434328479553028099v5] [rootPath=by-dev/meta/queryCoord-segmentMeta/434328479553028099] [2022/07/07 18:05:58.568 +08:00] [INFO] [shard_cluster_service.go:80] ["successfully add shard cluster"] [collectionID=434328479553028099] [replica=434327753245589530] [vchan=by-dev-rootcoord-dml_29_434328479553028099v5] [2022/07/07 18:05:58.568 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434328479553028100] [collectionID=434328479553028099] [partitions="[434328479553028100]"] [2022/07/07 18:05:58.568 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434328479553028100] [2022/07/07 18:05:58.568 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434328479906922499] [collectionID=434328479553028099] [partitions="[434328479553028100,434328479906922499]"] [2022/07/07 18:05:58.568 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434328479906922499] [2022/07/07 18:05:58.568 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434337916566634497] [collectionID=434328479553028099] [partitions="[434328479553028100,434328479906922499,434337916566634497]"] [2022/07/07 18:05:58.568 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434337916566634497] [2022/07/07 18:05:58.568 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434360565926199297] [collectionID=434328479553028099] [partitions="[434328479553028100,434328479906922499,434337916566634497,434360565926199297]"] [2022/07/07 18:05:58.569 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434360565926199297] [2022/07/07 18:05:58.569 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434383215154692098] [collectionID=434328479553028099] [partitions="[434328479553028100,434328479906922499,434337916566634497,434360565926199297,434383215154692098]"] [2022/07/07 18:05:58.569 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434383215154692098] [2022/07/07 18:05:58.569 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434405864160362498] [collectionID=434328479553028099] [partitions="[434328479553028100,434328479906922499,434337916566634497,434360565926199297,434383215154692098,434405864160362498]"] [2022/07/07 18:05:58.569 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434405864160362498] [2022/07/07 18:05:58.569 +08:00] [INFO] [task.go:218] ["loading growing segments in WatchDmChannels..."] [collectionID=434328479553028099] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:58.569 +08:00] [INFO] [task.go:227] ["successfully load growing segments done in WatchDmChannels"] [collectionID=434328479553028099] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:58.569 +08:00] [INFO] [task.go:259] ["watchDMChannel, group channels done"] [collectionID=434328479553028099] [2022/07/07 18:05:58.569 +08:00] [INFO] [task.go:274] ["watchDMChannel, add check points info for unflushed segments done"] [collectionID=434328479553028099] [unflushedSegmentIDs="[]"] [2022/07/07 18:05:58.569 +08:00] [INFO] [task.go:299] ["watchDMChannel, add check points info for flushed segments done"] [collectionID=434328479553028099] [flushedSegmentIDs="[]"] [2022/07/07 18:05:58.569 +08:00] [INFO] [task.go:324] ["watchDMChannel, add check points info for dropped segments done"] [collectionID=434328479553028099] [droppedSegmentIDs="[]"] [2022/07/07 18:05:58.569 +08:00] [INFO] [data_sync_service.go:99] ["add DML flow graph"] [collectionID=434328479553028099] [channel=by-dev-rootcoord-dml_29_434328479553028099v5] [2022/07/07 18:05:58.569 +08:00] [INFO] [task.go:335] ["Query node add DML flow graphs"] [collectionID=434328479553028099] [channels="[by-dev-rootcoord-dml_29_434328479553028099v5]"] [2022/07/07 18:05:58.574 +08:00] [INFO] [mq_msgstream.go:871] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_29] [MessageID={}] [2022/07/07 18:05:58.688 +08:00] [INFO] [mq_msgstream.go:877] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_29] [2022/07/07 18:05:58.700 +08:00] [INFO] [flow_graph_query_node.go:237] ["query node flow graph seeks from pChannel"] [collectionID=434328479553028099] [channel=by-dev-rootcoord-dml_29] [2022/07/07 18:05:58.701 +08:00] [INFO] [task.go:373] ["watchDMChannel, add flowGraph for dmChannels success"] [collectionID=434328479553028099] [vChannels="[by-dev-rootcoord-dml_29_434328479553028099v5]"] [2022/07/07 18:05:58.701 +08:00] [INFO] [collection.go:118] ["add vChannel to collection"] [collectionID=434328479553028099] [vChannel=by-dev-rootcoord-dml_29_434328479553028099v5] [2022/07/07 18:05:58.701 +08:00] [INFO] [collection.go:171] ["add pChannel to collection"] [collectionID=434328479553028099] [pChannel=by-dev-rootcoord-dml_29] [2022/07/07 18:05:58.701 +08:00] [INFO] [task.go:379] ["watchDMChannel, init replica done"] [collectionID=434328479553028099] [vChannels="[by-dev-rootcoord-dml_29_434328479553028099v5]"] [2022/07/07 18:05:58.701 +08:00] [INFO] [tsafe_replica.go:121] ["add tSafe done"] [channel=by-dev-rootcoord-dml_29_434328479553028099v5] [2022/07/07 18:05:58.701 +08:00] [INFO] [query_shard_service.go:96] ["Successfully add query shard"] [collection=434328479553028099] [replica=434327753245589530] [channel=by-dev-rootcoord-dml_29_434328479553028099v5] [2022/07/07 18:05:58.701 +08:00] [INFO] [task.go:398] ["WatchDmChannels done"] [collectionID=434328479553028099] [vChannels="[by-dev-rootcoord-dml_29_434328479553028099v5]"] [2022/07/07 18:05:58.701 +08:00] [INFO] [impl.go:134] ["watchDmChannelsTask WaitToFinish done"] [collectionID=434328479553028099] [nodeID=26] [2022/07/07 18:05:58.706 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:58.709 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434337948128772104] [state=3] [2022/07/07 18:05:58.709 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_18_434328479553028097v0] [segment="online_segments:<segmentID:434337948128772104 collectionID:434328479553028097 partitionID:434328479906922500 num_rows:26608 dmChannel:"by-dev-rootcoord-dml_18_434328479553028097v0" compactionFrom:434335133252714497 compactionFrom:434335085542506499 compactionFrom:434336029234692100 compactionFrom:434336972966199298 segment_state:Sealed replica_ids:434327753245589538 node_ids:26 > offline_segments:<segmentID:434337948128772104 collectionID:434328479553028097 partitionID:434328479906922500 num_rows:26608 dmChannel:"by-dev-rootcoord-dml_18_434328479553028097v0" compactionFrom:434335133252714497 compactionFrom:434335085542506499 compactionFrom:434336029234692100 compactionFrom:434336972966199298 segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.710 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434359622417514500] [state=3] [2022/07/07 18:05:58.710 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_23_434328479553028097v5] [segment="online_segments:<segmentID:434359622417514500 collectionID:434328479553028097 partitionID:434337916592848898 num_rows:4321 dmChannel:"by-dev-rootcoord-dml_23_434328479553028097v5" segment_state:Sealed replica_ids:434327753245589538 node_ids:26 > offline_segments:<segmentID:434359622417514500 collectionID:434328479553028097 partitionID:434337916592848898 num_rows:4321 dmChannel:"by-dev-rootcoord-dml_23_434328479553028097v5" segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.712 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434381422089732104] [state=3] [2022/07/07 18:05:58.712 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_18_434328479553028097v0] [segment="online_segments:<segmentID:434381422089732104 collectionID:434328479553028097 partitionID:434360565899984899 num_rows:53520 dmChannel:"by-dev-rootcoord-dml_18_434328479553028097v0" compactionFrom:434378590934532104 compactionFrom:434379440451485697 compactionFrom:434380384288112641 compactionFrom:434381327822749701 segment_state:Sealed replica_ids:434327753245589538 node_ids:26 > offline_segments:<segmentID:434381422089732104 collectionID:434328479553028097 partitionID:434360565899984899 num_rows:53520 dmChannel:"by-dev-rootcoord-dml_18_434328479553028097v0" compactionFrom:434378590934532104 compactionFrom:434379440451485697 compactionFrom:434380384288112641 compactionFrom:434381327822749701 segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.718 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.718 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.718 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.718 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.719 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.719 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.719 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.719 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.719 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.719 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.720 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.720 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.729 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801550] [collectionID=434328479553028099] [error="ShardCluster for by-dev-rootcoord-dml_29_434328479553028099v5 replicaID 434327753245589530 is no available"] [2022/07/07 18:05:58.730 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801551] [collectionID=434328479553028099] [error="ShardCluster for by-dev-rootcoord-dml_29_434328479553028099v5 replicaID 434327753245589530 is no available"] [2022/07/07 18:05:58.733 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"=null] [state=3] [2022/07/07 18:05:58.733 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_6_434328479448170498v0] [distribution=null] [2022/07/07 18:05:58.737 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"=null] [state=3] [2022/07/07 18:05:58.737 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_29_434328479553028099v5] [distribution=null] [2022/07/07 18:05:58.741 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"="[{"node_id":26,"partition_id":434337916592848898,"segment_ids":[434359622417514500]},{"node_id":26,"partition_id":434383215141584897,"segment_ids":[434404920612356097]},{"node_id":26,"partition_id":434405864160362497,"segment_ids":[434421957334138884]},{"node_id":27,"partition_id":434328479906922500,"segment_ids":[434337948888989697]},{"node_id":27,"partition_id":434337916592848898,"segment_ids":[434358757119492106]},{"node_id":27,"partition_id":434360565899984899,"segment_ids":[434381423544631297,434382271632900098]},{"node_id":27,"partition_id":434383215141584897,"segment_ids":[434404071331332100]}]"] [state=3] [2022/07/07 18:05:58.741 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_23_434328479553028097v5] [distribution="[{"node_id":26,"partition_id":434337916592848898,"segment_ids":[434359622417514500]},{"node_id":26,"partition_id":434383215141584897,"segment_ids":[434404920612356097]},{"node_id":26,"partition_id":434405864160362497,"segment_ids":[434421957334138884]},{"node_id":27,"partition_id":434328479906922500,"segment_ids":[434337948888989697]},{"node_id":27,"partition_id":434337916592848898,"segment_ids":[434358757119492106]},{"node_id":27,"partition_id":434360565899984899,"segment_ids":[434381423544631297,434382271632900098]},{"node_id":27,"partition_id":434383215141584897,"segment_ids":[434404071331332100]}]"] [2022/07/07 18:05:58.742 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"="[{"node_id":26,"partition_id":434328479906922500,"segment_ids":[434337948128772104]},{"node_id":26,"partition_id":434337916592848898,"segment_ids":[434359622417514498]},{"node_id":26,"partition_id":434360565899984899,"segment_ids":[434381422089732104]},{"node_id":26,"partition_id":434383215141584897,"segment_ids":[434404055602692102]},{"node_id":26,"partition_id":434405864160362497,"segment_ids":[434421957334138891]},{"node_id":27,"partition_id":434337916592848898,"segment_ids":[434358758548176897]},{"node_id":27,"partition_id":434360565899984899,"segment_ids":[434382271619792898]},{"node_id":27,"partition_id":434383215141584897,"segment_ids":[434403977025028098,434404920599248898]}]"] [state=3] [2022/07/07 18:05:58.742 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_18_434328479553028097v0] [distribution="[{"node_id":26,"partition_id":434328479906922500,"segment_ids":[434337948128772104]},{"node_id":26,"partition_id":434337916592848898,"segment_ids":[434359622417514498]},{"node_id":26,"partition_id":434360565899984899,"segment_ids":[434381422089732104]},{"node_id":26,"partition_id":434383215141584897,"segment_ids":[434404055602692102]},{"node_id":26,"partition_id":434405864160362497,"segment_ids":[434421957334138891]},{"node_id":27,"partition_id":434337916592848898,"segment_ids":[434358758548176897]},{"node_id":27,"partition_id":434360565899984899,"segment_ids":[434382271619792898]},{"node_id":27,"partition_id":434383215141584897,"segment_ids":[434403977025028098,434404920599248898]}]"] [2022/07/07 18:05:59.777 +08:00] [ERROR] [client.go:233] ["ClientBase Call grpc first call get error"] [role=querynode] [error="err: rpc error: code = Canceled desc = context canceled\n, /opt/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/opt/milvus/internal/util/grpcclient/client.go:232 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n/opt/milvus/internal/distributed/querynode/client/client.go:227 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/opt/milvus/internal/querynode/shard_cluster.go:781 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n/usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n\t/opt/milvus/internal/util/grpcclient/client.go:233\ngithub.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n\t/opt/milvus/internal/distributed/querynode/client/client.go:227\ngithub.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n\t/opt/milvus/internal/querynode/shard_cluster.go:781"] [2022/07/07 18:05:59.777 +08:00] [ERROR] [shard_cluster.go:798] ["Search 27 failed, reason err err: rpc error: code = Canceled desc = context canceled\n, /opt/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/opt/milvus/internal/util/grpcclient/client.go:232 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n/opt/milvus/internal/distributed/querynode/client/client.go:227 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/opt/milvus/internal/querynode/shard_cluster.go:781 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n/usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search\n\t/opt/milvus/internal/querynode/shard_cluster.go:798\ngithub.com/milvus-io/milvus/internal/querynode.(*QueryNode).searchWithDmlChannel\n\t/opt/milvus/internal/querynode/impl.go:615\ngithub.com/milvus-io/milvus/internal/querynode.(*QueryNode).Search.func1\n\t/opt/milvus/internal/querynode/impl.go:463\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/gopath/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]

Jul 13 '22 07:07 yhmo

partial log of query node 002:

[2022/07/07 18:05:42.513 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.513 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.514 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.514 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.515 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.515 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.515 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.515 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.515 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.516 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.516 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.516 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.517 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.517 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.872 +08:00] [INFO] [impl.go:229] ["loadSegmentsTask Enqueue done"] [collectionID=434328479553028097] [segmentIDs="[434404920599248899,434404071331332102]"] [nodeID=27] [2022/07/07 18:05:42.872 +08:00] [INFO] [task.go:507] ["LoadSegmentTask PreExecute start"] [msgID=0] [2022/07/07 18:05:42.872 +08:00] [INFO] [task.go:533] ["LoadSegmentTask PreExecute done"] [msgID=0] [2022/07/07 18:05:42.872 +08:00] [INFO] [task.go:539] ["LoadSegmentTask Execute start"] [msgID=0] [2022/07/07 18:05:42.872 +08:00] [INFO] [segment_loader.go:86] ["segmentLoader start loading..."] [collectionID=434328479553028097] [segmentNum=2] [segmentType=Sealed] [2022/07/07 18:05:42.873 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404920599248899] [segmentType=Sealed] [2022/07/07 18:05:42.873 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404071331332102] [segmentType=Sealed] [2022/07/07 18:05:42.873 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404071331332102] [segmentType=Sealed] [2022/07/07 18:05:42.873 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404920599248899] [segmentType=Sealed] [2022/07/07 18:05:42.984 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434404920599248899] [fieldID=102] [2022/07/07 18:05:42.984 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434404920599248899] [fieldID=102] [2022/07/07 18:05:42.999 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=3853] [segmentID=434404920599248899] [2022/07/07 18:05:42.999 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=3853] [segmentID=434404920599248899] [2022/07/07 18:05:42.999 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=3853] [segmentID=434404920599248899] [2022/07/07 18:05:42.999 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=3853] [segmentID=434404920599248899] [2022/07/07 18:05:42.999 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434404920599248899] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:43.036 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434404920599248899] [2022/07/07 18:05:43.089 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434404071331332102] [fieldID=102] [2022/07/07 18:05:43.089 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434328479553028097] [segment=434404071331332102] [fieldID=102] [2022/07/07 18:05:43.110 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=51951] [segmentID=434404071331332102] [2022/07/07 18:05:43.141 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=51951] [segmentID=434404071331332102] [2022/07/07 18:05:43.141 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=51951] [segmentID=434404071331332102] [2022/07/07 18:05:43.142 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=101] ["row count"=51951] [segmentID=434404071331332102] [2022/07/07 18:05:43.142 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434328479553028097] [segment=434404071331332102] [len(field)=4] [segmentType=Sealed] [2022/07/07 18:05:43.368 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434404071331332102] [2022/07/07 18:05:43.388 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404071331332102] [segmentType=Sealed] [2022/07/07 18:05:43.388 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028097] [partitionID=434383215141584897] [segmentID=434404920599248899] [segmentType=Sealed] [2022/07/07 18:05:43.388 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_19_434328479553028097v1" msgID:"\010\275\224\021\020\000\030\000 \000" "] ["msg id"="CL2UERAAGAAgAA=="] [2022/07/07 18:05:43.393 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_19] [subname=querynode-27-434328479553028097] [2022/07/07 18:05:43.401 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_19] [MessageID={}] [2022/07/07 18:05:43.513 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_19] [2022/07/07 18:05:43.513 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_19] [2022/07/07 18:05:46.364 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434328479553028097] [channel=by-dev-rootcoord-dml_19] ["msg id"="CL2UERAAGAAgAA=="] [2022/07/07 18:05:46.364 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"="CL2UERAAGAAgAA=="] [2022/07/07 18:05:46.372 +08:00] [INFO] [task.go:558] ["LoadSegmentTask Execute done"] [msgID=0] [2022/07/07 18:05:46.372 +08:00] [INFO] [impl.go:241] ["loadSegmentsTask WaitToFinish done"] [collectionID=434328479553028097] [segmentIDs="[434404920599248899,434404071331332102]"] [nodeID=27] [2022/07/07 18:05:48.795 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800238] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:48.795 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800238] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:48.890 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800242] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:48.890 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800242] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:49.075 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800269] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:49.075 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800269] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:49.360 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800290] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:49.360 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800290] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.338 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800418] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.338 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800418] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.530 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800432] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.530 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800432] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.536 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800436] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.536 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800436] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.542 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800438] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.542 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800438] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.599 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800450] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.599 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800450] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.613 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800455] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:50.613 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800455] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.100 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800493] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.100 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800493] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.113 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800500] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.113 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800500] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.738 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800583] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.738 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800583] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.739 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800584] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.739 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800584] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.749 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800600] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.749 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800600] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.749 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800601] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.749 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800601] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.783 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800617] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.783 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800617] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.783 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800618] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:51.783 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800618] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:52.344 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800656] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:52.344 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800656] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:52.366 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800671] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:52.366 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800671] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:52.950 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800725] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:52.950 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800725] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:52.950 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800726] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:52.950 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800726] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.005 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800743] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.005 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800743] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.006 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800744] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.006 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800744] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.195 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800778] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.195 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800778] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.195 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800777] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.195 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800777] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.348 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800828] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.348 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800828] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.397 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800840] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.397 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800840] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.550 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800853] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.550 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800853] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.610 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800866] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.610 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800866] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.753 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800890] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.753 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800890] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.835 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800904] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.835 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800904] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.836 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800906] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.836 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800906] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.888 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800910] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.888 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800910] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.918 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800918] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.918 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800918] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.936 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800925] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:53.936 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800925] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.080 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800942] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.080 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800942] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.081 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800943] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.081 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800943] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.249 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800960] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.249 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800960] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.258 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800967] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.258 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800967] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.354 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801017] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.354 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801017] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.354 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801018] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.354 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801018] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.499 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801051] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.499 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801051] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.499 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801052] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:54.499 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801052] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.175 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801148] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.175 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801148] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.211 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801156] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.211 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801156] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.237 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801160] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.237 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801160] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.440 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801182] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.440 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801182] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.445 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801185] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.445 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801185] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.575 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801202] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.575 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801202] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.579 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801204] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.579 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801204] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.592 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801208] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.592 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801208] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.688 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801219] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.688 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801219] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.699 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801224] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.699 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801224] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.736 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801228] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.736 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801228] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.762 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801232] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.762 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801232] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.977 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801244] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:55.977 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801244] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.100 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801269] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.100 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801269] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.289 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801278] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.289 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801278] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.312 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801285] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.312 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801285] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.312 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801286] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.312 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801286] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.356 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801298] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.356 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801298] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.593 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801348] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.593 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801348] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.593 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801349] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.593 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801349] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.598 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801353] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.598 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801353] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.599 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801354] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:56.599 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801354] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.110 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801394] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.110 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801394] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.111 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801395] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.111 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801395] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.248 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801405] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.248 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801405] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.249 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801406] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.249 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801406] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.298 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801419] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.298 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801419] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.383 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801432] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.383 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801432] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.399 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801439] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.399 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801439] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.465 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801444] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.465 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801444] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.617 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801472] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.617 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801472] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.680 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801500] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:57.680 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801500] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.097 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801535] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.097 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801535] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.109 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801541] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.109 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801541] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.158 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801555] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.158 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801555] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.283 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801564] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.283 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801564] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.295 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801570] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.295 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801570] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.466 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801596] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_13_434328479540183041v1 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.466 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801596] [collectionID=434328479540183041] [error="ShardCluster for by-dev-rootcoord-dml_16_434328479540183041v4 replicaID 434327753245589522 is no available"] [2022/07/07 18:05:58.706 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:58.707 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434337948128772102] [state=3] [2022/07/07 18:05:58.707 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_20_434328479553028097v2] [segment="online_segments:<segmentID:434337948128772102 collectionID:434328479553028097 partitionID:434328479906922500 num_rows:26732 dmChannel:"by-dev-rootcoord-dml_20_434328479553028097v2" compactionFrom:434335132702474251 compactionFrom:434335085542506498 compactionFrom:434336029247799297 compactionFrom:434336972966199297 segment_state:Sealed replica_ids:434327753245589538 node_ids:26 > offline_segments:<segmentID:434337948128772102 collectionID:434328479553028097 partitionID:434328479906922500 num_rows:26732 dmChannel:"by-dev-rootcoord-dml_20_434328479553028097v2" compactionFrom:434335132702474251 compactionFrom:434335085542506498 compactionFrom:434336029247799297 compactionFrom:434336972966199297 segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.712 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434381422089732100] [state=3] [2022/07/07 18:05:58.712 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_20_434328479553028097v2] [segment="online_segments:<segmentID:434381422089732100 collectionID:434328479553028097 partitionID:434360565899984899 num_rows:53405 dmChannel:"by-dev-rootcoord-dml_20_434328479553028097v2" compactionFrom:434378590934532100 compactionFrom:434379440451485698 compactionFrom:434380384261898241 compactionFrom:434381327822749700 segment_state:Sealed replica_ids:434327753245589538 node_ids:26 > offline_segments:<segmentID:434381422089732100 collectionID:434328479553028097 partitionID:434360565899984899 num_rows:53405 dmChannel:"by-dev-rootcoord-dml_20_434328479553028097v2" compactionFrom:434378590934532100 compactionFrom:434379440451485698 compactionFrom:434380384261898241 compactionFrom:434381327822749700 segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.713 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434404071331332106] [state=3] [2022/07/07 18:05:58.713 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_20_434328479553028097v2] [segment="online_segments:<segmentID:434404071331332106 collectionID:434328479553028097 partitionID:434383215141584897 num_rows:51976 dmChannel:"by-dev-rootcoord-dml_20_434328479553028097v2" compactionFrom:434401240176132106 compactionFrom:434402089640656897 compactionFrom:434403033411485699 compactionFrom:434403977025028099 segment_state:Sealed replica_ids:434327753245589538 node_ids:26 > offline_segments:<segmentID:434404071331332106 collectionID:434328479553028097 partitionID:434383215141584897 num_rows:51976 dmChannel:"by-dev-rootcoord-dml_20_434328479553028097v2" compactionFrom:434401240176132106 compactionFrom:434402089640656897 compactionFrom:434403033411485699 compactionFrom:434403977025028099 segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.717 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.717 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.718 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.718 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.718 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.718 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.719 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.719 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.720 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.720 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.720 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.720 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.721 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.721 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.728 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"=null] [state=3] [2022/07/07 18:05:58.728 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_0_434328479435325441v0] [distribution=null] [2022/07/07 18:05:58.729 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801550] [collectionID=434328479553028099] [error="ShardCluster for by-dev-rootcoord-dml_28_434328479553028099v4 replicaID 434327753245589530 is no available"] [2022/07/07 18:05:58.729 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801551] [collectionID=434328479553028099] [error="ShardCluster for by-dev-rootcoord-dml_28_434328479553028099v4 replicaID 434327753245589530 is no available"] [2022/07/07 18:05:58.730 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"=null] [state=3] [2022/07/07 18:05:58.730 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_4_434328479435325441v4] [distribution=null] [2022/07/07 18:05:58.734 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"=null] [state=3] [2022/07/07 18:05:58.734 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_16_434328479540183041v4] [distribution=null] [2022/07/07 18:05:58.735 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"=null] [state=3] [2022/07/07 18:05:58.735 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_13_434328479540183041v1] [distribution=null] [2022/07/07 18:05:58.736 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"=null] [state=3] [2022/07/07 18:05:58.736 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_28_434328479553028099v4] [distribution=null] [2022/07/07 18:05:58.739 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"="[{"node_id":26,"partition_id":434328479906922500,"segment_ids":[434337948128772102]},{"node_id":26,"partition_id":434360565899984899,"segment_ids":[434381422089732100]},{"node_id":26,"partition_id":434383215141584897,"segment_ids":[434404071331332106]},{"node_id":26,"partition_id":434405864160362497,"segment_ids":[434421958212583425]},{"node_id":27,"partition_id":434337916592848898,"segment_ids":[434359622430621697,434358757119492102]},{"node_id":27,"partition_id":434360565899984899,"segment_ids":[434382271619792897]},{"node_id":27,"partition_id":434383215141584897,"segment_ids":[434404920599248900]}]"] [state=3] [2022/07/07 18:05:58.739 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_20_434328479553028097v2] [distribution="[{"node_id":26,"partition_id":434328479906922500,"segment_ids":[434337948128772102]},{"node_id":26,"partition_id":434360565899984899,"segment_ids":[434381422089732100]},{"node_id":26,"partition_id":434383215141584897,"segment_ids":[434404071331332106]},{"node_id":26,"partition_id":434405864160362497,"segment_ids":[434421958212583425]},{"node_id":27,"partition_id":434337916592848898,"segment_ids":[434359622430621697,434358757119492102]},{"node_id":27,"partition_id":434360565899984899,"segment_ids":[434382271619792897]},{"node_id":27,"partition_id":434383215141584897,"segment_ids":[434404920599248900]}]"] [2022/07/07 18:05:58.742 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"="[{"node_id":26,"partition_id":434328480012042241,"segment_ids":[434337932649168897]},{"node_id":26,"partition_id":434337916580003841,"segment_ids":[434358678450339841,434358725662212099,434359622247120897]},{"node_id":26,"partition_id":434360565899984898,"segment_ids":[434381406361354250,434382271554519041]},{"node_id":26,"partition_id":434383215115370498,"segment_ids":[434404920560189441,434404056048336897]},{"node_id":26,"partition_id":434405864134148097,"segment_ids":[434420964113776645,434421907592577025,434421026805252105]}]"] [state=3] [2022/07/07 18:05:58.742 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_34_434328479553028101v4] [distribution="[{"node_id":26,"partition_id":434328480012042241,"segment_ids":[434337932649168897]},{"node_id":26,"partition_id":434337916580003841,"segment_ids":[434358678450339841,434358725662212099,434359622247120897]},{"node_id":26,"partition_id":434360565899984898,"segment_ids":[434381406361354250,434382271554519041]},{"node_id":26,"partition_id":434383215115370498,"segment_ids":[434404920560189441,434404056048336897]},{"node_id":26,"partition_id":434405864134148097,"segment_ids":[434420964113776645,434421907592577025,434421026805252105]}]"] [2022/07/07 18:06:00.411 +08:00] [WARN] [task_scheduler.go:326] ["search context timeout"] [2022/07/07 18:06:00.411 +08:00] [ERROR] [shard_cluster.go:798] ["stream operation failed: search context timeout"] [stack="github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search\n\t/opt/milvus/internal/querynode/shard_cluster.go:798\ngithub.com/milvus-io/milvus/internal/querynode.(*QueryNode).searchWithDmlChannel\n\t/opt/milvus/internal/querynode/impl.go:615\ngithub.com/milvus-io/milvus/internal/querynode.(*QueryNode).Search.func1\n\t/opt/milvus/internal/querynode/impl.go:463\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/gopath/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"] [2022/07/07 18:06:00.411 +08:00] [ERROR] [shard_cluster.go:798] ["stream operation failed: search context timeout"] [stack="github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search\n\t/opt/milvus/internal/querynode/shard_cluster.go:798\ngithub.com/milvus-io/milvus/internal/querynode.(*QueryNode).searchWithDmlChannel\n\t/opt/milvus/internal/querynode/impl.go:615\ngithub.com/milvus-io/milvus/internal/querynode.(*QueryNode).Search.func1\n\t/opt/milvus/internal/querynode/impl.go:463\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/gopath/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"] [2022/07/07 18:06:00.411 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800736] [collectionID=434328479553028097] [error="stream operation failed: search context timeout"] [2022/07/07 18:06:00.411 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622800355] [collectionID=434328479553028097] [error="stream operation failed: search context timeout"] [2022/07/07 18:06:00.412 +08:00] [ERROR] [client.go:233] ["ClientBase Call grpc first call get error"] [role=querynode] [error="err: rpc error: code = Canceled desc = context canceled\n, /opt/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/opt/milvus/internal/util/grpcclient/client.go:232 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n/opt/milvus/internal/distributed/querynode/client/client.go:227 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/opt/milvus/internal/querynode/shard_cluster.go:781 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n/usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n\t/opt/milvus/internal/util/grpcclient/client.go:233\ngithub.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n\t/opt/milvus/internal/distributed/querynode/client/client.go:227\ngithub.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n\t/opt/milvus/internal/querynode/shard_cluster.go:781"] [2022/07/07 18:06:00.412 +08:00] [ERROR] [client.go:233] ["ClientBase Call grpc first call get error"] [role=querynode] [error="err: rpc error: code = Canceled desc = context canceled\n, /opt/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/opt/milvus/internal/util/grpcclient/client.go:232 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n/opt/milvus/internal/distributed/querynode/client/client.go:227 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/opt/milvus/internal/querynode/shard_cluster.go:781 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n/usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n\t/opt/milvus/internal/util/grpcclient/client.go:233\ngithub.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n\t/opt/milvus/internal/distributed/querynode/client/client.go:227\ngithub.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n\t/opt/milvus/internal/querynode/shard_cluster.go:781"]

Jul 13 '22 07:07 yhmo

partial log of query node 003:

[2022/07/07 18:04:09.996 +08:00] [INFO] [logutil.go:165] ["Log directory"] [configDir=] [2022/07/07 18:04:09.996 +08:00] [INFO] [logutil.go:166] ["Set log file to "] [path=/home/logs/milvus/logs/querynode-29.log] [2022/07/07 18:04:09.996 +08:00] [INFO] [query_node.go:147] ["QueryNode init session"] [nodeID=29] ["node address"=10.97.104.209:21123] [2022/07/07 18:04:10.000 +08:00] [INFO] [minio_chunk_manager.go:105] ["minio chunk manager init success."] [bucketname=a-bucket] [root=/var/lib/milvus/data/] [2022/07/07 18:04:10.000 +08:00] [INFO] [query_node.go:231] ["queryNode try to connect etcd success"] [MetaRootPath=by-dev/meta] [2022/07/07 18:04:10.001 +08:00] [INFO] [query_node.go:245] ["query node init successfully"] [queryNodeID=29] [IP=10.97.104.209] [Port=21123] [2022/07/07 18:04:10.001 +08:00] [INFO] [query_node.go:317] ["query node watchChangeInfo start"] [2022/07/07 18:04:10.004 +08:00] [INFO] [minio_chunk_manager.go:105] ["minio chunk manager init success."] [bucketname=a-bucket] [root=/var/lib/milvus/data/] [2022/07/07 18:04:10.004 +08:00] [INFO] [query_node.go:274] ["query node start successfully"] [queryNodeID=29] [IP=10.97.104.209] [Port=21123] [2022/07/07 18:04:10.006 +08:00] [INFO] [session_util.go:241] ["Service registered successfully"] [ServerName=querynode] [serverID=29] [2022/07/07 18:04:39.067 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:04:39.284 +08:00] [INFO] [impl.go:123] ["watchDmChannelsTask Enqueue done"] [collectionID=434328479553028101] [nodeID=29] [replicaID=434327753245589546] [2022/07/07 18:04:39.284 +08:00] [INFO] [task.go:146] ["Starting WatchDmChannels ..."] [collectionName=verificationbaseimage] [collectionID=434328479553028101] [replicaID=434327753245589546] ["load type"=LoadCollection] [vChannels="[by-dev-rootcoord-dml_30_434328479553028101v0]"] [pChannels="[by-dev-rootcoord-dml_30]"] [2022/07/07 18:04:39.284 +08:00] [INFO] [impl.go:123] ["watchDmChannelsTask Enqueue done"] [collectionID=434328479553028097] [nodeID=29] [replicaID=434327753245589538] [2022/07/07 18:04:39.287 +08:00] [INFO] [collection.go:329] ["create collection"] [collectionID=434328479553028101] [2022/07/07 18:04:39.287 +08:00] [INFO] [shard_node_detector.go:68] ["nodeDetector watch"] [collectionID=434328479553028101] [replicaID=434327753245589546] [vchannelName=by-dev-rootcoord-dml_30_434328479553028101v0] [2022/07/07 18:04:39.289 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=12] [2022/07/07 18:04:39.289 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=13] [2022/07/07 18:04:39.289 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=9] [2022/07/07 18:04:39.289 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=10] [2022/07/07 18:04:39.289 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=11] [2022/07/07 18:04:39.289 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=14] [2022/07/07 18:04:39.290 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=29] [2022/07/07 18:04:39.290 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=26] [2022/07/07 18:04:39.292 +08:00] [INFO] [base_table.go:462] ["Pulsar config"] ["pulsar url"=pulsar://hadoop2045.rz.momo.com:6650,hadoop2047.rz.momo.com:6650,hadoop2048.rz.momo.com:6650] ["pulsar web url"=http://hadoop2048.rz.momo.com:8080] [2022/07/07 18:04:39.293 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=30] [2022/07/07 18:04:39.293 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=27] [2022/07/07 18:04:39.293 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=31] [2022/07/07 18:04:39.293 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=28] [2022/07/07 18:04:39.293 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=32] [2022/07/07 18:04:39.293 +08:00] [INFO] [shard_segment_detector.go:74] ["segmentDetector start watch"] [collectionID=434328479553028101] [replicaID=434327753245589546] [vchannelName=by-dev-rootcoord-dml_30_434328479553028101v0] [rootPath=by-dev/meta/queryCoord-segmentMeta/434328479553028101] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434337932400394246] [state=3] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434358741390852102] [state=3] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434381327796535298] [state=3] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434381390632452099] [state=3] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434382271567364098] [state=3] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434404056441552897] [state=3] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434404920573034498] [state=3] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434420964113776643] [state=3] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434421026805252107] [state=3] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434421907579469825] [state=3] [2022/07/07 18:04:39.296 +08:00] [INFO] [shard_cluster_service.go:80] ["successfully add shard cluster"] [collectionID=434328479553028101] [replica=434327753245589546] [vchan=by-dev-rootcoord-dml_30_434328479553028101v0] [2022/07/07 18:04:39.296 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434328479553028102] [collectionID=434328479553028101] [partitions="[434328479553028102]"] [2022/07/07 18:04:39.296 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434328479553028102] [2022/07/07 18:04:39.296 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434328480012042241] [collectionID=434328479553028101] [partitions="[434328479553028102,434328480012042241]"] [2022/07/07 18:04:39.296 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434328480012042241] [2022/07/07 18:04:39.296 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434337916580003841] [collectionID=434328479553028101] [partitions="[434328479553028102,434328480012042241,434337916580003841]"] [2022/07/07 18:04:39.296 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434337916580003841] [2022/07/07 18:04:39.296 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434360565899984898] [collectionID=434328479553028101] [partitions="[434328479553028102,434328480012042241,434337916580003841,434360565899984898]"] [2022/07/07 18:04:39.296 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434360565899984898] [2022/07/07 18:04:39.296 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434383215115370498] [collectionID=434328479553028101] [partitions="[434328479553028102,434328480012042241,434337916580003841,434360565899984898,434383215115370498]"] [2022/07/07 18:04:39.296 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434383215115370498] [2022/07/07 18:04:39.296 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434405864134148097] [collectionID=434328479553028101] [partitions="[434328479553028102,434328480012042241,434337916580003841,434360565899984898,434383215115370498,434405864134148097]"] [2022/07/07 18:04:39.296 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434405864134148097] [2022/07/07 18:04:39.296 +08:00] [INFO] [task.go:218] ["loading growing segments in WatchDmChannels..."] [collectionID=434328479553028101] [unFlushedSegmentIDs="[]"] [2022/07/07 18:04:39.296 +08:00] [INFO] [task.go:227] ["successfully load growing segments done in WatchDmChannels"] [collectionID=434328479553028101] [unFlushedSegmentIDs="[]"] [2022/07/07 18:04:39.296 +08:00] [INFO] [task.go:259] ["watchDMChannel, group channels done"] [collectionID=434328479553028101] [2022/07/07 18:04:39.296 +08:00] [INFO] [task.go:274] ["watchDMChannel, add check points info for unflushed segments done"] [collectionID=434328479553028101] [unflushedSegmentIDs="[]"] [2022/07/07 18:04:39.297 +08:00] [INFO] [task.go:299] ["watchDMChannel, add check points info for flushed segments done"] [collectionID=434328479553028101] [flushedSegmentIDs="[434337932400394246,434358741390852102,434359622247120898,434382271567364098,434381327796535298,434381390632452099,434404056441552897,434404920573034498,434420964113776643,434421026805252107,434421907579469825]"] [2022/07/07 18:04:39.297 +08:00] [INFO] [task.go:324] ["watchDMChannel, add check points info for dropped segments done"] [collectionID=434328479553028101] [droppedSegmentIDs="[434333198158397444,434332270089994244,434329423245213698,434330366924292097,434332254479056900,434334141784784897,434328480103530499,434336972900663298,434331310669168641,434335116973572102,434336029169156099,434335085451018243,434357734600605697,434351128821104642,434339803911684097,434350184840298500,434347353895075842,434337916724183042,434341691296055298,434356791013539842,434350232196874247,434349241252970501,434338860337463297,434352103905034245,434353959740112898,434345466208976899,434344554275536897,434341707273994248,434358678463447041,434353015995498499,434340747708727299,434355847164067842,434346409940746241,434352072368848897,434354903812669446,434343578877296641,434346473051652100,434344522529898500,434348297639428098,434355894507274244,434342635381456897,434375712593412099,434365284452925443,434366228013776898,434368115699613698,434361509500420100,434363396858576897,434377552870768642,434371890586583043,434364340747370497,434375665486135297,434367203399172100,434362453415428097,434369075107332099,434378496444727301,434360566031056900,434373778036490241,434374721662877697,434376609165213697,434380384117719042,434369059431120902,434370946631991301,434372834317828100,434372865709572099,434367171706224642,434364356620189697,434379440241770497,434370003070877697,434378559477252099,434398314767056899,434399258262896642,434396427094327301,434395483428356097,434395562399498241,434398393292554250,434400202125213697,434384158728912898,434403976946647044,434389884504375297,434392715423645697,434394539709956099,434387933694263297,434386109224714246,434392652351799300,434386046244618242,434397370956906500,434393595860746241,434390764810665985,434391708672983042,434389821091741700,434401145765232642,434383215233335300,434401224447492100,434403033424855041,434388877294698501,434386989949648898,434402089522692102,434385102578384898,434411526496976898,434412470464413699,434405864278327297,434417188990877697,434418195650052106,434413414195920899,434420020054589441,434407751885520901,434419076480368646,434409639243677701,434408695289548100,434410582909911043,434409654998794250,434412501882372100,434406807983620097,434415348766474244,434415301672042500,434418132774813699,434416245193834502,434414357901213698]"] [2022/07/07 18:04:39.297 +08:00] [INFO] [data_sync_service.go:99] ["add DML flow graph"] [collectionID=434328479553028101] [channel=by-dev-rootcoord-dml_30_434328479553028101v0] [2022/07/07 18:04:39.297 +08:00] [INFO] [task.go:335] ["Query node add DML flow graphs"] [collectionID=434328479553028101] [channels="[by-dev-rootcoord-dml_30_434328479553028101v0]"] [2022/07/07 18:04:39.305 +08:00] [INFO] [mq_msgstream.go:871] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_30] [MessageID={}] [2022/07/07 18:04:39.411 +08:00] [INFO] [mq_msgstream.go:877] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_30] [2022/07/07 18:04:39.427 +08:00] [INFO] [flow_graph_query_node.go:237] ["query node flow graph seeks from pChannel"] [collectionID=434328479553028101] [channel=by-dev-rootcoord-dml_30] [2022/07/07 18:04:39.427 +08:00] [INFO] [task.go:373] ["watchDMChannel, add flowGraph for dmChannels success"] [collectionID=434328479553028101] [vChannels="[by-dev-rootcoord-dml_30_434328479553028101v0]"] [2022/07/07 18:04:39.427 +08:00] [INFO] [collection.go:118] ["add vChannel to collection"] [collectionID=434328479553028101] [vChannel=by-dev-rootcoord-dml_30_434328479553028101v0] [2022/07/07 18:04:39.427 +08:00] [INFO] [collection.go:171] ["add pChannel to collection"] [collectionID=434328479553028101] [pChannel=by-dev-rootcoord-dml_30] [2022/07/07 18:04:39.427 +08:00] [INFO] [task.go:379] ["watchDMChannel, init replica done"] [collectionID=434328479553028101] [vChannels="[by-dev-rootcoord-dml_30_434328479553028101v0]"] [2022/07/07 18:04:39.427 +08:00] [INFO] [tsafe_replica.go:121] ["add tSafe done"] [channel=by-dev-rootcoord-dml_30_434328479553028101v0] [2022/07/07 18:04:39.427 +08:00] [INFO] [query_shard_service.go:96] ["Successfully add query shard"] [collection=434328479553028101] [replica=434327753245589546] [channel=by-dev-rootcoord-dml_30_434328479553028101v0] [2022/07/07 18:04:39.427 +08:00] [INFO] [task.go:398] ["WatchDmChannels done"] [collectionID=434328479553028101] [vChannels="[by-dev-rootcoord-dml_30_434328479553028101v0]"] [2022/07/07 18:04:39.427 +08:00] [INFO] [task.go:146] ["Starting WatchDmChannels ..."] [collectionName=useralbumfirstpicture] [collectionID=434328479553028097] [replicaID=434327753245589538] ["load type"=LoadCollection] [vChannels="[by-dev-rootcoord-dml_19_434328479553028097v1]"] [pChannels="[by-dev-rootcoord-dml_19]"] [2022/07/07 18:04:39.427 +08:00] [INFO] [impl.go:134] ["watchDmChannelsTask WaitToFinish done"] [collectionID=434328479553028101] [nodeID=29] [2022/07/07 18:04:39.427 +08:00] [INFO] [collection.go:329] ["create collection"] [collectionID=434328479553028097] [2022/07/07 18:04:39.427 +08:00] [INFO] [shard_node_detector.go:68] ["nodeDetector watch"] [collectionID=434328479553028097] [replicaID=434327753245589538] [vchannelName=by-dev-rootcoord-dml_19_434328479553028097v1] [2022/07/07 18:04:39.429 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=13] [2022/07/07 18:04:39.429 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=9] [2022/07/07 18:04:39.429 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=14] [2022/07/07 18:04:39.429 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=10] [2022/07/07 18:04:39.429 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=11] [2022/07/07 18:04:39.429 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=12] [2022/07/07 18:04:39.429 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=31] [2022/07/07 18:04:39.429 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=32] [2022/07/07 18:04:39.429 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=30] [2022/07/07 18:04:39.429 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=26] [2022/07/07 18:04:39.429 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=27] [2022/07/07 18:04:39.429 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=28] [2022/07/07 18:04:39.429 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=29] [2022/07/07 18:04:39.429 +08:00] [INFO] [shard_segment_detector.go:74] ["segmentDetector start watch"] [collectionID=434328479553028097] [replicaID=434327753245589538] [vchannelName=by-dev-rootcoord-dml_19_434328479553028097v1] [rootPath=by-dev/meta/queryCoord-segmentMeta/434328479553028097] [2022/07/07 18:04:39.431 +08:00] [INFO] [shard_cluster_service.go:80] ["successfully add shard cluster"] [collectionID=434328479553028097] [replica=434327753245589538] [vchan=by-dev-rootcoord-dml_19_434328479553028097v1] [2022/07/07 18:04:39.431 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434328479553028098] [collectionID=434328479553028097] [partitions="[434328479553028098]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434328479553028098] [2022/07/07 18:04:39.431 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434328479906922500] [collectionID=434328479553028097] [partitions="[434328479553028098,434328479906922500]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434328479906922500] [2022/07/07 18:04:39.431 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434337916592848898] [collectionID=434328479553028097] [partitions="[434328479553028098,434328479906922500,434337916592848898]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434337916592848898] [2022/07/07 18:04:39.431 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434360565899984899] [collectionID=434328479553028097] [partitions="[434328479553028098,434328479906922500,434337916592848898,434360565899984899]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434360565899984899] [2022/07/07 18:04:39.431 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434383215141584897] [collectionID=434328479553028097] [partitions="[434328479553028098,434328479906922500,434337916592848898,434360565899984899,434383215141584897]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434383215141584897] [2022/07/07 18:04:39.431 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434405864160362497] [collectionID=434328479553028097] [partitions="[434328479553028098,434328479906922500,434337916592848898,434360565899984899,434383215141584897,434405864160362497]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434405864160362497] [2022/07/07 18:04:39.431 +08:00] [INFO] [task.go:218] ["loading growing segments in WatchDmChannels..."] [collectionID=434328479553028097] [unFlushedSegmentIDs="[]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [task.go:227] ["successfully load growing segments done in WatchDmChannels"] [collectionID=434328479553028097] [unFlushedSegmentIDs="[]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [task.go:259] ["watchDMChannel, group channels done"] [collectionID=434328479553028097] [2022/07/07 18:04:39.431 +08:00] [INFO] [task.go:274] ["watchDMChannel, add check points info for unflushed segments done"] [collectionID=434328479553028097] [unflushedSegmentIDs="[]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [task.go:299] ["watchDMChannel, add check points info for flushed segments done"] [collectionID=434328479553028097] [flushedSegmentIDs="[434337948128772106,434358757119492104,434359622417514499,434382271632900097,434381422089732106,434404920599248899,434404071331332102,434421957334138889]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [task.go:324] ["watchDMChannel, add check points info for dropped segments done"] [collectionID=434328479553028097] [droppedSegmentIDs="[434336029234692098,434331310695383044,434329423350071298,434336972953354241,434330367016304643,434335085542506497,434328480103530497,434335132702474246,434332285818372104,434333198223671299,434334141968285697,434332254505271299,434354038527754244,434343578916356097,434356791065706497,434348297678749697,434341723002634244,434344585614852100,434339804056125442,434347353973456897,434353016139939841,434348360488452099,434354903956848641,434337916763504641,434345466326941698,434350184905834499,434357734692618243,434346410123984897,434355847268925441,434340747853168642,434341691427127298,434352072539242497,434358678502768643,434350247925514248,434344522529898505,434338860508119042,434351128938807298,434353959831863297,434342635302813697,434355910589808641,434349241318506497,434370003215319041,434367171797712898,434364340904919041,434375665591255043,434376609204797441,434363396924112897,434371009730052104,434378591996477441,434362453507178500,434375744050954244,434374721794211843,434379440438378498,434380384274743297,434377553093328897,434369059496656898,434360566188343297,434371890809405441,434381327822749698,434367234856452100,434364372244234244,434372834462007297,434370946697789441,434366228118634497,434373778062442499,434365284596842500,434361509605539841,434368115817578499,434378496484048900,434372897403043841,434392652417597441,434394539827920897,434387933838704643,434403977038135297,434399258354384898,434386046336106497,434401145856720897,434392730982154246,434389884150480900,434384158925783042,434395483467677697,434389821249290245,434402089654026245,434396427330256897,434395577865732102,434398409925591041,434400202230071297,434386093495812102,434398314832592897,434385102630813697,434393596004663299,434403033398640642,434397371009335297,434388877334020097,434390764993904641,434386990015184897,434391708672983047,434401241250922497,434383215338192897,434420020067434500,434412470490628099,434418132840349699,434414358084976643,434412533339652099,434417189056413701,434419076585226242,434411526576144387,434405864448720898,434410582831005699,434416245272739844,434421907684065282,434415301803114499,434420964205264899,434419139971383297,434413414300778499,434415348766474250,434407751951056898,434408695433527298,434409639282999298,434406808232919043,434409670727172102]"] [2022/07/07 18:04:39.431 +08:00] [INFO] [data_sync_service.go:99] ["add DML flow graph"] [collectionID=434328479553028097] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [2022/07/07 18:04:39.431 +08:00] [INFO] [task.go:335] ["Query node add DML flow graphs"] [collectionID=434328479553028097] [channels="[by-dev-rootcoord-dml_19_434328479553028097v1]"] [2022/07/07 18:04:39.437 +08:00] [INFO] [mq_msgstream.go:871] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_19] [MessageID={}] [2022/07/07 18:04:39.558 +08:00] [INFO] [mq_msgstream.go:877] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_19] [2022/07/07 18:04:39.573 +08:00] [INFO] [flow_graph_query_node.go:237] ["query node flow graph seeks from pChannel"] [collectionID=434328479553028097] [channel=by-dev-rootcoord-dml_19] [2022/07/07 18:04:39.573 +08:00] [INFO] [task.go:373] ["watchDMChannel, add flowGraph for dmChannels success"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_19_434328479553028097v1]"] [2022/07/07 18:04:39.573 +08:00] [INFO] [collection.go:118] ["add vChannel to collection"] [collectionID=434328479553028097] [vChannel=by-dev-rootcoord-dml_19_434328479553028097v1] [2022/07/07 18:04:39.573 +08:00] [INFO] [collection.go:171] ["add pChannel to collection"] [collectionID=434328479553028097] [pChannel=by-dev-rootcoord-dml_19] [2022/07/07 18:04:39.573 +08:00] [INFO] [task.go:379] ["watchDMChannel, init replica done"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_19_434328479553028097v1]"] [2022/07/07 18:04:39.573 +08:00] [INFO] [tsafe_replica.go:121] ["add tSafe done"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [2022/07/07 18:04:39.573 +08:00] [INFO] [query_shard_service.go:96] ["Successfully add query shard"] [collection=434328479553028097] [replica=434327753245589538] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [2022/07/07 18:04:39.573 +08:00] [INFO] [task.go:398] ["WatchDmChannels done"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_19_434328479553028097v1]"] [2022/07/07 18:04:39.573 +08:00] [INFO] [impl.go:134] ["watchDmChannelsTask WaitToFinish done"] [collectionID=434328479553028097] [nodeID=29] [2022/07/07 18:04:54.842 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:04:54.844 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434358757119492104] [state=3] [2022/07/07 18:04:54.844 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [segment="online_segments:<segmentID:434358757119492104 collectionID:434328479553028097 partitionID:434337916592848898 num_rows:50872 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" compactionFrom:434355910589808641 compactionFrom:434356791065706497 compactionFrom:434357734692618243 compactionFrom:434358678502768643 segment_state:Sealed replica_ids:434327753245589538 node_ids:27 > offline_segments:<segmentID:434358757119492104 collectionID:434328479553028097 partitionID:434337916592848898 num_rows:50872 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" compactionFrom:434355910589808641 compactionFrom:434356791065706497 compactionFrom:434357734692618243 compactionFrom:434358678502768643 segment_state:Sealed replica_ids:434327753245589538 node_ids:13 > "] [2022/07/07 18:04:54.849 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434382271632900097] [state=3] [2022/07/07 18:04:54.849 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [segment="online_segments:<segmentID:434382271632900097 collectionID:434328479553028097 partitionID:434360565899984899 num_rows:4497 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" segment_state:Sealed replica_ids:434327753245589538 node_ids:27 > offline_segments:<segmentID:434382271632900097 collectionID:434328479553028097 partitionID:434360565899984899 num_rows:4497 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" segment_state:Sealed replica_ids:434327753245589538 node_ids:13 > "] [2022/07/07 18:04:54.855 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=13] [2022/07/07 18:04:54.855 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=13] [addr=] [2022/07/07 18:04:54.857 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=13] [2022/07/07 18:04:54.857 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=13] [addr=] [2022/07/07 18:04:55.506 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:04:55.507 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434421957334138889] [state=3] [2022/07/07 18:04:55.507 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [segment="online_segments:<segmentID:434421957334138889 collectionID:434328479553028097 partitionID:434405864160362497 num_rows:33639 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" compactionFrom:434419139971383297 compactionFrom:434421907684065282 compactionFrom:434419076585226242 compactionFrom:434420020067434500 compactionFrom:434420964205264899 segment_state:Sealed replica_ids:434327753245589538 node_ids:27 > "] [2022/07/07 18:04:55.507 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [segment="offline_segments:<segmentID:434419139971383297 collectionID:434328479553028097 partitionID:434405864160362497 num_rows:22572 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" compactionFrom:434415348766474250 compactionFrom:434416245272739844 compactionFrom:434417189056413701 compactionFrom:434418132840349699 segment_state:Sealed replica_ids:434327753245589538 node_ids:11 > offline_segments:<segmentID:434419076585226242 collectionID:434328479553028097 partitionID:434405864160362497 num_rows:2679 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" segment_state:Sealed replica_ids:434327753245589538 node_ids:11 > offline_segments:<segmentID:434420020067434500 collectionID:434328479553028097 partitionID:434405864160362497 num_rows:2817 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" segment_state:Sealed replica_ids:434327753245589538 node_ids:11 > offline_segments:<segmentID:434420964205264899 collectionID:434328479553028097 partitionID:434405864160362497 num_rows:2905 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" segment_state:Sealed replica_ids:434327753245589538 node_ids:11 > "] [2022/07/07 18:04:55.508 +08:00] [INFO] [shard_cluster.go:470] ["No node of event is in cluster, skip to process it"] [nodes="[11]"] [2022/07/07 18:04:55.508 +08:00] [INFO] [shard_cluster.go:470] ["No node of event is in cluster, skip to process it"] [nodes="[11]"] [2022/07/07 18:04:55.509 +08:00] [INFO] [shard_cluster.go:470] ["No node of event is in cluster, skip to process it"] [nodes="[11]"] [2022/07/07 18:04:55.510 +08:00] [INFO] [shard_cluster.go:470] ["No node of event is in cluster, skip to process it"] [nodes="[11]"] [2022/07/07 18:05:22.327 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:22.333 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434381422089732106] [state=3] [2022/07/07 18:05:22.333 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [segment="online_segments:<segmentID:434381422089732106 collectionID:434328479553028097 partitionID:434360565899984899 num_rows:53803 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" compactionFrom:434378591996477441 compactionFrom:434379440438378498 compactionFrom:434380384274743297 compactionFrom:434381327822749698 segment_state:Sealed replica_ids:434327753245589538 node_ids:27 > offline_segments:<segmentID:434381422089732106 collectionID:434328479553028097 partitionID:434360565899984899 num_rows:53803 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" compactionFrom:434378591996477441 compactionFrom:434379440438378498 compactionFrom:434380384274743297 compactionFrom:434381327822749698 segment_state:Sealed replica_ids:434327753245589538 node_ids:11 > "] [2022/07/07 18:05:22.340 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=11] [2022/07/07 18:05:22.340 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=11] [addr=] [2022/07/07 18:05:22.342 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=11] [2022/07/07 18:05:22.342 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=11] [addr=] [2022/07/07 18:05:23.033 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:23.557 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=14] [2022/07/07 18:05:23.557 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=14] [addr=] [2022/07/07 18:05:23.559 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=14] [2022/07/07 18:05:23.559 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=14] [addr=] [2022/07/07 18:05:24.182 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:24.801 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:24.915 +08:00] [INFO] [impl.go:123] ["watchDmChannelsTask Enqueue done"] [collectionID=434328479435325441] [nodeID=29] [replicaID=434327753245589506] [2022/07/07 18:05:24.915 +08:00] [INFO] [task.go:146] ["Starting WatchDmChannels ..."] [collectionName=famousstarimage] [collectionID=434328479435325441] [replicaID=434327753245589506] ["load type"=LoadCollection] [vChannels="[by-dev-rootcoord-dml_1_434328479435325441v1]"] [pChannels="[by-dev-rootcoord-dml_1]"] [2022/07/07 18:05:24.915 +08:00] [INFO] [collection.go:329] ["create collection"] [collectionID=434328479435325441] [2022/07/07 18:05:24.915 +08:00] [INFO] [shard_node_detector.go:68] ["nodeDetector watch"] [collectionID=434328479435325441] [replicaID=434327753245589506] [vchannelName=by-dev-rootcoord-dml_1_434328479435325441v1] [2022/07/07 18:05:24.917 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=10] [2022/07/07 18:05:24.917 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=12] [2022/07/07 18:05:24.917 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=9] [2022/07/07 18:05:24.917 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=29] [2022/07/07 18:05:24.917 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=26] [2022/07/07 18:05:24.917 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=32] [2022/07/07 18:05:24.917 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=31] [2022/07/07 18:05:24.917 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=27] [2022/07/07 18:05:24.917 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=28] [2022/07/07 18:05:24.917 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=30] [2022/07/07 18:05:24.917 +08:00] [INFO] [shard_segment_detector.go:74] ["segmentDetector start watch"] [collectionID=434328479435325441] [replicaID=434327753245589506] [vchannelName=by-dev-rootcoord-dml_1_434328479435325441v1] [rootPath=by-dev/meta/queryCoord-segmentMeta/434328479435325441] [2022/07/07 18:05:24.918 +08:00] [INFO] [shard_cluster_service.go:80] ["successfully add shard cluster"] [collectionID=434328479435325441] [replica=434327753245589506] [vchan=by-dev-rootcoord-dml_1_434328479435325441v1] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434328479435325442] [collectionID=434328479435325441] [partitions="[434328479435325442]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434328479435325442] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434328479566135297] [collectionID=434328479435325441] [partitions="[434328479435325442,434328479566135297]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434328479566135297] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434337916592848897] [collectionID=434328479435325441] [partitions="[434328479435325442,434328479566135297,434337916592848897]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434337916592848897] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434360565913092099] [collectionID=434328479435325441] [partitions="[434328479435325442,434328479566135297,434337916592848897,434360565913092099]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434360565913092099] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434383215154692097] [collectionID=434328479435325441] [partitions="[434328479435325442,434328479566135297,434337916592848897,434360565913092099,434383215154692097]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434383215154692097] [2022/07/07 18:05:24.918 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434405864134148098] [collectionID=434328479435325441] [partitions="[434328479435325442,434328479566135297,434337916592848897,434360565913092099,434383215154692097,434405864134148098]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434405864134148098] [2022/07/07 18:05:24.918 +08:00] [INFO] [task.go:218] ["loading growing segments in WatchDmChannels..."] [collectionID=434328479435325441] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [task.go:227] ["successfully load growing segments done in WatchDmChannels"] [collectionID=434328479435325441] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [task.go:259] ["watchDMChannel, group channels done"] [collectionID=434328479435325441] [2022/07/07 18:05:24.918 +08:00] [INFO] [task.go:274] ["watchDMChannel, add check points info for unflushed segments done"] [collectionID=434328479435325441] [unflushedSegmentIDs="[]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [task.go:299] ["watchDMChannel, add check points info for flushed segments done"] [collectionID=434328479435325441] [flushedSegmentIDs="[]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [task.go:324] ["watchDMChannel, add check points info for dropped segments done"] [collectionID=434328479435325441] [droppedSegmentIDs="[]"] [2022/07/07 18:05:24.918 +08:00] [INFO] [data_sync_service.go:99] ["add DML flow graph"] [collectionID=434328479435325441] [channel=by-dev-rootcoord-dml_1_434328479435325441v1] [2022/07/07 18:05:24.918 +08:00] [INFO] [task.go:335] ["Query node add DML flow graphs"] [collectionID=434328479435325441] [channels="[by-dev-rootcoord-dml_1_434328479435325441v1]"] [2022/07/07 18:05:24.923 +08:00] [INFO] [mq_msgstream.go:871] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_1] [MessageID={}] [2022/07/07 18:05:24.924 +08:00] [INFO] [impl.go:123] ["watchDmChannelsTask Enqueue done"] [collectionID=434328479553028097] [nodeID=29] [replicaID=434327753245589538] [2022/07/07 18:05:25.028 +08:00] [INFO] [mq_msgstream.go:877] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_1] [2022/07/07 18:05:25.038 +08:00] [INFO] [flow_graph_query_node.go:237] ["query node flow graph seeks from pChannel"] [collectionID=434328479435325441] [channel=by-dev-rootcoord-dml_1] [2022/07/07 18:05:25.038 +08:00] [INFO] [task.go:373] ["watchDMChannel, add flowGraph for dmChannels success"] [collectionID=434328479435325441] [vChannels="[by-dev-rootcoord-dml_1_434328479435325441v1]"] [2022/07/07 18:05:25.038 +08:00] [INFO] [collection.go:118] ["add vChannel to collection"] [collectionID=434328479435325441] [vChannel=by-dev-rootcoord-dml_1_434328479435325441v1] [2022/07/07 18:05:25.038 +08:00] [INFO] [collection.go:171] ["add pChannel to collection"] [collectionID=434328479435325441] [pChannel=by-dev-rootcoord-dml_1] [2022/07/07 18:05:25.038 +08:00] [INFO] [task.go:379] ["watchDMChannel, init replica done"] [collectionID=434328479435325441] [vChannels="[by-dev-rootcoord-dml_1_434328479435325441v1]"] [2022/07/07 18:05:25.038 +08:00] [INFO] [tsafe_replica.go:121] ["add tSafe done"] [channel=by-dev-rootcoord-dml_1_434328479435325441v1] [2022/07/07 18:05:25.038 +08:00] [INFO] [query_shard_service.go:96] ["Successfully add query shard"] [collection=434328479435325441] [replica=434327753245589506] [channel=by-dev-rootcoord-dml_1_434328479435325441v1] [2022/07/07 18:05:25.038 +08:00] [INFO] [task.go:398] ["WatchDmChannels done"] [collectionID=434328479435325441] [vChannels="[by-dev-rootcoord-dml_1_434328479435325441v1]"] [2022/07/07 18:05:25.038 +08:00] [INFO] [task.go:146] ["Starting WatchDmChannels ..."] [collectionName=useralbumfirstpicture] [collectionID=434328479553028097] [replicaID=434327753245589538] ["load type"=LoadCollection] [vChannels="[by-dev-rootcoord-dml_22_434328479553028097v4]"] [pChannels="[by-dev-rootcoord-dml_22]"] [2022/07/07 18:05:25.038 +08:00] [INFO] [shard_node_detector.go:68] ["nodeDetector watch"] [collectionID=434328479553028097] [replicaID=434327753245589538] [vchannelName=by-dev-rootcoord-dml_22_434328479553028097v4] [2022/07/07 18:05:25.038 +08:00] [INFO] [impl.go:134] ["watchDmChannelsTask WaitToFinish done"] [collectionID=434328479435325441] [nodeID=29] [2022/07/07 18:05:25.040 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=12] [2022/07/07 18:05:25.040 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=9] [2022/07/07 18:05:25.040 +08:00] [WARN] [shard_node_detector.go:104] ["Node not found in session"] ["node id"=10] [2022/07/07 18:05:25.040 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=26] [2022/07/07 18:05:25.040 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=30] [2022/07/07 18:05:25.040 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=27] [2022/07/07 18:05:25.040 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=31] [2022/07/07 18:05:25.040 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=29] [2022/07/07 18:05:25.040 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=28] [2022/07/07 18:05:25.040 +08:00] [INFO] [shard_cluster.go:201] ["ShardCluster add node"] [nodeID=32] [2022/07/07 18:05:25.040 +08:00] [INFO] [shard_segment_detector.go:74] ["segmentDetector start watch"] [collectionID=434328479553028097] [replicaID=434327753245589538] [vchannelName=by-dev-rootcoord-dml_22_434328479553028097v4] [rootPath=by-dev/meta/queryCoord-segmentMeta/434328479553028097] [2022/07/07 18:05:25.041 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434337948128772100] [state=3] [2022/07/07 18:05:25.041 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434358757119492100] [state=3] [2022/07/07 18:05:25.041 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434359622404407297] [state=3] [2022/07/07 18:05:25.041 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434382271632900099] [state=3] [2022/07/07 18:05:25.041 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434404920599248897] [state=3] [2022/07/07 18:05:25.041 +08:00] [INFO] [shard_cluster_service.go:80] ["successfully add shard cluster"] [collectionID=434328479553028097] [replica=434327753245589538] [vchan=by-dev-rootcoord-dml_22_434328479553028097v4] [2022/07/07 18:05:25.041 +08:00] [INFO] [task.go:218] ["loading growing segments in WatchDmChannels..."] [collectionID=434328479553028097] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:25.041 +08:00] [INFO] [task.go:227] ["successfully load growing segments done in WatchDmChannels"] [collectionID=434328479553028097] [unFlushedSegmentIDs="[]"] [2022/07/07 18:05:25.041 +08:00] [INFO] [task.go:259] ["watchDMChannel, group channels done"] [collectionID=434328479553028097] [2022/07/07 18:05:25.041 +08:00] [INFO] [task.go:274] ["watchDMChannel, add check points info for unflushed segments done"] [collectionID=434328479553028097] [unflushedSegmentIDs="[]"] [2022/07/07 18:05:25.041 +08:00] [INFO] [task.go:299] ["watchDMChannel, add check points info for flushed segments done"] [collectionID=434328479553028097] [flushedSegmentIDs="[434337948128772100,434359622404407297,434358757119492100,434382271632900099,434381422089732102,434404072615837697,434404920599248897,434421957334138886]"] [2022/07/07 18:05:25.042 +08:00] [INFO] [task.go:324] ["watchDMChannel, add check points info for dropped segments done"] [collectionID=434328479553028097] [droppedSegmentIDs="[434336972953354245,434334141968285698,434332286093623297,434333198210826242,434336029234692101,434335085555613697,434335132702474244,434332254518378497,434330367016304642,434328480090685442,434329423350071297,434331310682013697,434346410137354241,434339804056125441,434342635315920899,434337916750397441,434356791052861442,434353959831863299,434338860520964099,434347353973456898,434352072539242501,434351128938807299,434348297665642499,434350248751267841,434344569886212103,434358678502768644,434349241331613700,434345466314096641,434354903956848642,434344522543005697,434346457323274244,434340747840061443,434354038527754248,434343578929463298,434357734692618242,434341723002634248,434355847255818241,434355910235652107,434353016139939843,434350184905834498,434341691414020098,434375665577885699,434377553093328900,434363396898160642,434372897835319297,434378496484048898,434380384274743298,434366228118634501,434378590934532106,434367171797712897,434364372244234248,434360566162391042,434362453507178498,434370003228164097,434381327835856897,434369059496656897,434368115830947841,434376609204797443,434367219127812105,434371890809405443,434379440451485699,434365284596842499,434370946710634498,434371010241232897,434372834448900099,434375744050954246,434364340904919044,434361509605539842,434373778062442498,434374721780842499,434398409020932102,434392730982154244,434401145869828098,434400202230071298,434399258341539841,434393595991556097,434391708659613699,434401240176132102,434397371022442497,434403033411485698,434386093495812104,434389821249290241,434387933851549698,434396427356471297,434395578573520897,434403977038135298,434388877320912898,434384158912413699,434395483454832643,434392652417597443,434402089654026241,434386990015184899,434386046349213698,434398314845700099,434383215364669442,434385102630813699,434394539827920898,434389884150480897,434390764993904645,434412517716131841,434414358084976641,434419139368452106,434413414313885697,434418132840349697,434416245272739843,434409670989578241,434411526576144386,434410582909911042,434409639282999300,434419076572056897,434415301790007297,434405864448720897,434406808219811841,434415364494852100,434407751964164099,434408695433527297,434420020067434499,434412470503735299,434421907697434626,434420964205264900,434417189056413699]"] [2022/07/07 18:05:25.042 +08:00] [INFO] [data_sync_service.go:99] ["add DML flow graph"] [collectionID=434328479553028097] [channel=by-dev-rootcoord-dml_22_434328479553028097v4] [2022/07/07 18:05:25.042 +08:00] [INFO] [task.go:335] ["Query node add DML flow graphs"] [collectionID=434328479553028097] [channels="[by-dev-rootcoord-dml_22_434328479553028097v4]"] [2022/07/07 18:05:25.047 +08:00] [INFO] [mq_msgstream.go:871] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_22] [MessageID={}] [2022/07/07 18:05:25.166 +08:00] [INFO] [mq_msgstream.go:877] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_22] [2022/07/07 18:05:25.175 +08:00] [INFO] [flow_graph_query_node.go:237] ["query node flow graph seeks from pChannel"] [collectionID=434328479553028097] [channel=by-dev-rootcoord-dml_22] [2022/07/07 18:05:25.175 +08:00] [INFO] [task.go:373] ["watchDMChannel, add flowGraph for dmChannels success"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_22_434328479553028097v4]"] [2022/07/07 18:05:25.175 +08:00] [INFO] [collection.go:118] ["add vChannel to collection"] [collectionID=434328479553028097] [vChannel=by-dev-rootcoord-dml_22_434328479553028097v4] [2022/07/07 18:05:25.176 +08:00] [INFO] [collection.go:171] ["add pChannel to collection"] [collectionID=434328479553028097] [pChannel=by-dev-rootcoord-dml_22] [2022/07/07 18:05:25.176 +08:00] [INFO] [task.go:379] ["watchDMChannel, init replica done"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_22_434328479553028097v4]"] [2022/07/07 18:05:25.176 +08:00] [INFO] [tsafe_replica.go:121] ["add tSafe done"] [channel=by-dev-rootcoord-dml_22_434328479553028097v4] [2022/07/07 18:05:25.176 +08:00] [INFO] [query_shard_service.go:96] ["Successfully add query shard"] [collection=434328479553028097] [replica=434327753245589538] [channel=by-dev-rootcoord-dml_22_434328479553028097v4] [2022/07/07 18:05:25.176 +08:00] [INFO] [task.go:398] ["WatchDmChannels done"] [collectionID=434328479553028097] [vChannels="[by-dev-rootcoord-dml_22_434328479553028097v4]"] [2022/07/07 18:05:25.176 +08:00] [INFO] [impl.go:134] ["watchDmChannelsTask WaitToFinish done"] [collectionID=434328479553028097] [nodeID=29] [2022/07/07 18:05:29.709 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:29.711 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434404072615837697] [state=3] [2022/07/07 18:05:29.711 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_22_434328479553028097v4] [segment="online_segments:<segmentID:434404072615837697 collectionID:434328479553028097 partitionID:434383215141584897 num_rows:51605 dmChannel:"by-dev-rootcoord-dml_22_434328479553028097v4" compactionFrom:434401240176132102 compactionFrom:434402089654026241 compactionFrom:434403033411485698 compactionFrom:434403977038135298 segment_state:Sealed replica_ids:434327753245589538 node_ids:26 > offline_segments:<segmentID:434404072615837697 collectionID:434328479553028097 partitionID:434383215141584897 num_rows:51605 dmChannel:"by-dev-rootcoord-dml_22_434328479553028097v4" compactionFrom:434401240176132102 compactionFrom:434402089654026241 compactionFrom:434403033411485698 compactionFrom:434403977038135298 segment_state:Sealed replica_ids:434327753245589538 node_ids:12 > "] [2022/07/07 18:05:29.713 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=12] [2022/07/07 18:05:29.713 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=12] [addr=] [2022/07/07 18:05:29.714 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=12] [2022/07/07 18:05:29.714 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=12] [addr=] [2022/07/07 18:05:29.714 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=12] [2022/07/07 18:05:29.714 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=12] [addr=] [2022/07/07 18:05:29.714 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=12] [2022/07/07 18:05:29.714 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=12] [addr=] [2022/07/07 18:05:29.724 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"=null] [state=3] [2022/07/07 18:05:29.724 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_1_434328479435325441v1] [distribution=null] [2022/07/07 18:05:42.507 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:42.510 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434359622247120898] [state=3] [2022/07/07 18:05:42.510 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_30_434328479553028101v0] [segment="online_segments:<segmentID:434359622247120898 collectionID:434328479553028101 partitionID:434337916580003841 num_rows:859 dmChannel:"by-dev-rootcoord-dml_30_434328479553028101v0" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > offline_segments:<segmentID:434359622247120898 collectionID:434328479553028101 partitionID:434337916580003841 num_rows:859 dmChannel:"by-dev-rootcoord-dml_30_434328479553028101v0" segment_state:Sealed replica_ids:434327753245589546 node_ids:9 > "] [2022/07/07 18:05:42.513 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.513 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.514 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.514 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.514 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.514 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:42.516 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=9] [2022/07/07 18:05:42.516 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=9] [addr=] [2022/07/07 18:05:58.706 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 18:05:58.708 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434337948128772106] [state=3] [2022/07/07 18:05:58.708 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [segment="online_segments:<segmentID:434337948128772106 collectionID:434328479553028097 partitionID:434328479906922500 num_rows:26810 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" compactionFrom:434335132702474246 compactionFrom:434335085542506497 compactionFrom:434336972953354241 compactionFrom:434336029234692098 segment_state:Sealed replica_ids:434327753245589538 node_ids:26 > offline_segments:<segmentID:434337948128772106 collectionID:434328479553028097 partitionID:434328479906922500 num_rows:26810 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" compactionFrom:434335132702474246 compactionFrom:434335085542506497 compactionFrom:434336972953354241 compactionFrom:434336029234692098 segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.710 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434359622417514499] [state=3] [2022/07/07 18:05:58.710 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [segment="online_segments:<segmentID:434359622417514499 collectionID:434328479553028097 partitionID:434337916592848898 num_rows:4346 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" segment_state:Sealed replica_ids:434327753245589538 node_ids:26 > offline_segments:<segmentID:434359622417514499 collectionID:434328479553028097 partitionID:434337916592848898 num_rows:4346 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.713 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434381422089732102] [state=3] [2022/07/07 18:05:58.713 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_22_434328479553028097v4] [segment="online_segments:<segmentID:434381422089732102 collectionID:434328479553028097 partitionID:434360565899984899 num_rows:53330 dmChannel:"by-dev-rootcoord-dml_22_434328479553028097v4" compactionFrom:434378590934532106 compactionFrom:434379440451485699 compactionFrom:434380384274743298 compactionFrom:434381327835856897 segment_state:Sealed replica_ids:434327753245589538 node_ids:26 > offline_segments:<segmentID:434381422089732102 collectionID:434328479553028097 partitionID:434360565899984899 num_rows:53330 dmChannel:"by-dev-rootcoord-dml_22_434328479553028097v4" compactionFrom:434378590934532106 compactionFrom:434379440451485699 compactionFrom:434380384274743298 compactionFrom:434381327835856897 segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.714 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434404920599248899] [state=3] [2022/07/07 18:05:58.714 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [segment="online_segments:<segmentID:434404920599248899 collectionID:434328479553028097 partitionID:434383215141584897 num_rows:3853 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" segment_state:Sealed replica_ids:434327753245589538 node_ids:27 > offline_segments:<segmentID:434404920599248899 collectionID:434328479553028097 partitionID:434383215141584897 num_rows:3853 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.714 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=27] [segmentID=434404071331332102] [state=3] [2022/07/07 18:05:58.714 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [segment="online_segments:<segmentID:434404071331332102 collectionID:434328479553028097 partitionID:434383215141584897 num_rows:51951 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" compactionFrom:434401241250922497 compactionFrom:434402089654026245 compactionFrom:434403033398640642 compactionFrom:434403977038135297 segment_state:Sealed replica_ids:434327753245589538 node_ids:27 > offline_segments:<segmentID:434404071331332102 collectionID:434328479553028097 partitionID:434383215141584897 num_rows:51951 dmChannel:"by-dev-rootcoord-dml_19_434328479553028097v1" compactionFrom:434401241250922497 compactionFrom:434402089654026245 compactionFrom:434403033398640642 compactionFrom:434403977038135297 segment_state:Sealed replica_ids:434327753245589538 node_ids:10 > "] [2022/07/07 18:05:58.717 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.717 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.718 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.718 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.718 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.718 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.720 +08:00] [INFO] [shard_cluster.go:227] ["ShardCluster remove node"] [nodeID=10] [2022/07/07 18:05:58.720 +08:00] [WARN] [shard_cluster.go:233] ["ShardCluster removeNode does not belong to it"] [nodeID=10] [addr=] [2022/07/07 18:05:58.728 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"=null] [state=3] [2022/07/07 18:05:58.728 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_1_434328479435325441v1] [distribution=null] [2022/07/07 18:05:58.739 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"="[{"node_id":26,"partition_id":434328479906922500,"segment_ids":[434337948128772106]},{"node_id":26,"partition_id":434337916592848898,"segment_ids":[434359622417514499]},{"node_id":27,"partition_id":434337916592848898,"segment_ids":[434358757119492104]},{"node_id":27,"partition_id":434360565899984899,"segment_ids":[434382271632900097,434381422089732106]},{"node_id":27,"partition_id":434383215141584897,"segment_ids":[434404920599248899,434404071331332102]},{"node_id":27,"partition_id":434405864160362497,"segment_ids":[434421957334138889]}]"] [state=3] [2022/07/07 18:05:58.739 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_19_434328479553028097v1] [distribution="[{"node_id":26,"partition_id":434328479906922500,"segment_ids":[434337948128772106]},{"node_id":26,"partition_id":434337916592848898,"segment_ids":[434359622417514499]},{"node_id":27,"partition_id":434337916592848898,"segment_ids":[434358757119492104]},{"node_id":27,"partition_id":434360565899984899,"segment_ids":[434382271632900097,434381422089732106]},{"node_id":27,"partition_id":434383215141584897,"segment_ids":[434404920599248899,434404071331332102]},{"node_id":27,"partition_id":434405864160362497,"segment_ids":[434421957334138889]}]"] [2022/07/07 18:05:58.740 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"="[{"node_id":-1,"partition_id":434405864160362497,"segment_ids":[434419139368452106,434420020067434499,434419076572056897,434420964205264900]},{"node_id":26,"partition_id":434360565899984899,"segment_ids":[434381422089732102]},{"node_id":26,"partition_id":434383215141584897,"segment_ids":[434404072615837697]},{"node_id":27,"partition_id":434328479906922500,"segment_ids":[434337948128772100]},{"node_id":27,"partition_id":434337916592848898,"segment_ids":[434358757119492100,434359622404407297]},{"node_id":27,"partition_id":434360565899984899,"segment_ids":[434382271632900099]},{"node_id":27,"partition_id":434383215141584897,"segment_ids":[434404920599248897]}]"] [state=3] [2022/07/07 18:05:58.740 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_22_434328479553028097v4] [distribution="[{"node_id":-1,"partition_id":434405864160362497,"segment_ids":[434419139368452106,434420020067434499,434419076572056897,434420964205264900]},{"node_id":26,"partition_id":434360565899984899,"segment_ids":[434381422089732102]},{"node_id":26,"partition_id":434383215141584897,"segment_ids":[434404072615837697]},{"node_id":27,"partition_id":434328479906922500,"segment_ids":[434337948128772100]},{"node_id":27,"partition_id":434337916592848898,"segment_ids":[434358757119492100,434359622404407297]},{"node_id":27,"partition_id":434360565899984899,"segment_ids":[434382271632900099]},{"node_id":27,"partition_id":434383215141584897,"segment_ids":[434404920599248897]}]"] [2022/07/07 18:05:58.743 +08:00] [INFO] [shard_cluster.go:279] ["ShardCluster sync segments"] ["replica segments"="[{"node_id":26,"partition_id":434328480012042241,"segment_ids":[434337932400394246]},{"node_id":26,"partition_id":434337916580003841,"segment_ids":[434359622247120898,434358741390852102]},{"node_id":26,"partition_id":434360565899984898,"segment_ids":[434381327796535298,434382271567364098,434381390632452099]},{"node_id":26,"partition_id":434383215115370498,"segment_ids":[434404920573034498,434404056441552897]},{"node_id":26,"partition_id":434405864134148097,"segment_ids":[434421026805252107,434421907579469825,434420964113776643]}]"] [state=3] [2022/07/07 18:05:58.743 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_30_434328479553028101v0] [distribution="[{"node_id":26,"partition_id":434328480012042241,"segment_ids":[434337932400394246]},{"node_id":26,"partition_id":434337916580003841,"segment_ids":[434359622247120898,434358741390852102]},{"node_id":26,"partition_id":434360565899984898,"segment_ids":[434381327796535298,434382271567364098,434381390632452099]},{"node_id":26,"partition_id":434383215115370498,"segment_ids":[434404920573034498,434404056441552897]},{"node_id":26,"partition_id":434405864134148097,"segment_ids":[434421026805252107,434421907579469825,434420964113776643]}]"] [2022/07/07 18:05:58.743 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801612] [collectionID=434328479553028097] [error="ShardCluster for by-dev-rootcoord-dml_22_434328479553028097v4 replicaID 434327753245589538 is no available"] [2022/07/07 18:05:58.743 +08:00] [WARN] [impl.go:617] ["search cluster failed"] [msgID=434421974622801611] [collectionID=434328479553028097] [error="ShardCluster for by-dev-rootcoord-dml_22_434328479553028097v4 replicaID 434327753245589538 is no available"] [2022/07/07 18:05:58.743 +08:00] [WARN] [task_scheduler.go:326] ["search context timeout"] [2022/07/07 18:05:58.743 +08:00] [WARN] [task_scheduler.go:326] ["search context timeout"] [2022/07/07 18:05:58.743 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: addrConn.createTransport failed to connect to {10.97.104.187:21123 10.97.104.187:21123 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp 10.97.104.187:21123: operation was canceled""] [2022/07/07 18:05:58.743 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: addrConn.createTransport failed to connect to {10.97.103.31:21123 10.97.103.31:21123 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp 10.97.103.31:21123: operation was canceled""] [2022/07/07 18:05:58.743 +08:00] [ERROR] [client.go:233] ["ClientBase Call grpc first call get error"] [role=querynode] [error="err: failed to connect 10.97.103.31:21123, reason: context canceled\n, /opt/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/opt/milvus/internal/util/grpcclient/client.go:232 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n/opt/milvus/internal/distributed/querynode/client/client.go:227 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/opt/milvus/internal/querynode/shard_cluster.go:781 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n/usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n\t/opt/milvus/internal/util/grpcclient/client.go:233\ngithub.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n\t/opt/milvus/internal/distributed/querynode/client/client.go:227\ngithub.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n\t/opt/milvus/internal/querynode/shard_cluster.go:781"] [2022/07/07 18:05:58.743 +08:00] [ERROR] [client.go:233] ["ClientBase Call grpc first call get error"] [role=querynode] [error="err: failed to connect 10.97.104.187:21123, reason: context canceled\n, /opt/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/opt/milvus/internal/util/grpcclient/client.go:232 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n/opt/milvus/internal/distributed/querynode/client/client.go:227 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/opt/milvus/internal/querynode/shard_cluster.go:781 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n/usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n\t/opt/milvus/internal/util/grpcclient/client.go:233\ngithub.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n\t/opt/milvus/internal/distributed/querynode/client/client.go:227\ngithub.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n\t/opt/milvus/internal/querynode/shard_cluster.go:781"]

Jul 13 '22 07:07 yhmo

partial log of query node 004:

[2022/07/07 20:01:10.716 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434423795094650881 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:677 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > "] [2022/07/07 20:01:11.215 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 20:01:11.703 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 20:01:12.250 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 20:01:12.810 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:04.130 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434405864134148097] [segmentID=434424738865479685] [segmentType=Growing] [2022/07/07 21:01:04.130 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434405864134148097] [segmentID=434424738865479685] [segmentType=Growing] [2022/07/07 21:01:06.828 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:07.620 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:08.148 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:08.937 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:09.698 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:10.329 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:10.972 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:11.426 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:11.881 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:12.297 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:12.297 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434424738865479685] [state=3] [2022/07/07 21:01:12.364 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434424738865479685 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:755 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > "] [2022/07/07 21:01:12.725 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:01:13.191 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:04:14.953 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:04:15.694 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:04:16.354 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:04:17.089 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:04:17.466 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:04:17.833 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:05:13.563 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:05:14.100 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:05:14.541 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:05:14.541 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434424788489338886] [state=3] [2022/07/07 21:05:14.541 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434424788489338886 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:10432 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434421026805252101 compactionFrom:434422851415834627 compactionFrom:434423795094650881 compactionFrom:434420964113776642 segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > "] [2022/07/07 21:05:14.541 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434421026805252101] [state=0] [2022/07/07 21:05:14.541 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434422851415834627] [state=0] [2022/07/07 21:05:14.541 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434423795094650881] [state=0] [2022/07/07 21:05:14.541 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434420964113776642] [state=0] [2022/07/07 21:05:14.542 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434421026805252101] [state=1] [2022/07/07 21:05:14.542 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=26] [segmentID=434421026805252101] [2022/07/07 21:05:14.543 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434422851415834627] [state=1] [2022/07/07 21:05:14.543 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=26] [segmentID=434422851415834627] [2022/07/07 21:05:14.543 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434423795094650881] [state=1] [2022/07/07 21:05:14.543 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=26] [segmentID=434423795094650881] [2022/07/07 21:05:14.544 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434420964113776642] [state=1] [2022/07/07 21:05:14.544 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=26] [segmentID=434420964113776642] [2022/07/07 21:05:14.544 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="offline_segments:<segmentID:434421026805252101 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:8340 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434418195650052104 compactionFrom:434418132774813697 compactionFrom:434419076480368643 compactionFrom:434420020041482244 segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > offline_segments:<segmentID:434422851415834627 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:635 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > offline_segments:<segmentID:434423795094650881 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:677 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > offline_segments:<segmentID:434420964113776642 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:780 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > "] [2022/07/07 21:05:14.890 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:05:15.276 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 21:06:17.459 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 22:01:04.330 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434405864134148097] [segmentID=434425682610356225] [segmentType=Growing] [2022/07/07 22:01:04.331 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434405864134148097] [segmentID=434425682610356225] [segmentType=Growing] [2022/07/07 22:01:07.485 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 22:01:08.702 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 22:01:09.713 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 22:01:10.727 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 22:01:11.210 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 22:01:11.781 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 22:01:12.311 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 22:01:12.780 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 22:01:13.315 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 22:01:13.316 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434425682610356225] [state=3] [2022/07/07 22:01:13.316 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434425682610356225 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:883 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > "] [2022/07/07 22:01:13.848 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:03.530 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434405864134148097] [segmentID=434426626158362625] [segmentType=Growing] [2022/07/07 23:01:03.530 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434405864134148097] [segmentID=434426626158362625] [segmentType=Growing] [2022/07/07 23:01:06.394 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:06.960 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:07.771 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:08.562 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:09.382 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:10.116 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:10.730 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:11.188 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:11.624 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:11.625 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434426626158362625] [state=3] [2022/07/07 23:01:11.625 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434426626158362625 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:759 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > "] [2022/07/07 23:01:12.111 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:12.569 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:13.085 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:14.213 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:14.893 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:15.577 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:16.166 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:16.795 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:01:17.496 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:02:14.577 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:02:16.185 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/07 23:02:17.055 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:01:03.930 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434405864134148097] [segmentID=434427569981620226] [segmentType=Growing] [2022/07/08 00:01:03.931 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434405864134148097] [segmentID=434427569981620226] [segmentType=Growing] [2022/07/08 00:01:06.870 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:01:06.871 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434426644783431682] [state=3] [2022/07/08 00:01:06.872 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434424788489338886] [state=1] [2022/07/08 00:01:06.873 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434424738865479685] [state=1] [2022/07/08 00:01:06.874 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434426626158362625] [state=1] [2022/07/08 00:01:06.874 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434425682610356225] [state=1] [2022/07/08 00:01:06.887 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434426644783431682 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:12829 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434424788489338886 compactionFrom:434424738865479685 compactionFrom:434426626158362625 compactionFrom:434425682610356225 segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > "] [2022/07/08 00:01:06.887 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434424788489338886] [state=0] [2022/07/08 00:01:06.887 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=26] [segmentID=434424788489338886] [2022/07/08 00:01:06.887 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434424738865479685] [state=0] [2022/07/08 00:01:06.887 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=26] [segmentID=434424738865479685] [2022/07/08 00:01:06.887 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434426626158362625] [state=0] [2022/07/08 00:01:06.887 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=26] [segmentID=434426626158362625] [2022/07/08 00:01:06.887 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=26] [segmentID=434425682610356225] [state=0] [2022/07/08 00:01:06.887 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=26] [segmentID=434425682610356225] [2022/07/08 00:01:06.890 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="offline_segments:<segmentID:434424788489338886 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:10432 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434421026805252101 compactionFrom:434422851415834627 compactionFrom:434423795094650881 compactionFrom:434420964113776642 segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > offline_segments:<segmentID:434424738865479685 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:755 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > offline_segments:<segmentID:434426626158362625 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:759 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > offline_segments:<segmentID:434425682610356225 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:883 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > "] [2022/07/08 00:01:07.414 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:01:08.189 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:01:08.773 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:01:09.337 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:01:09.844 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:01:09.845 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=26] [segmentID=434427569981620226] [state=3] [2022/07/08 00:01:09.845 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434427569981620226 collectionID:434328479553028101 partitionID:434405864134148097 num_rows:859 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:26 > "] [2022/07/08 00:01:10.345 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:01:10.787 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:04:14.220 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:04:15.016 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:04:15.701 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:04:16.488 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:04:17.250 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:04:17.901 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:05:14.125 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:05:14.620 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 00:06:18.348 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 01:01:03.330 +08:00] [WARN] [flow_graph_filter_dm_node.go:118] ["invalid message type in filterDmNode"] ["message type"=CreatePartition] [collection=434328479435325441] [channel=by-dev-rootcoord-dml_5_434328479435325441v5] [2022/07/08 01:01:03.330 +08:00] [WARN] [flow_graph_filter_dm_node.go:118] ["invalid message type in filterDmNode"] ["message type"=CreatePartition] [collection=434328479553028101] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [2022/07/08 01:01:03.330 +08:00] [WARN] [flow_graph_filter_dm_node.go:118] ["invalid message type in filterDmNode"] ["message type"=CreatePartition] [collection=434328479540183041] [channel=by-dev-rootcoord-dml_15_434328479540183041v3] [2022/07/08 01:01:03.330 +08:00] [WARN] [flow_graph_filter_dm_node.go:118] ["invalid message type in filterDmNode"] ["message type"=CreatePartition] [collection=434328479553028099] [channel=by-dev-rootcoord-dml_25_434328479553028099v1] [2022/07/08 01:01:03.330 +08:00] [WARN] [flow_graph_filter_dm_node.go:118] ["invalid message type in filterDmNode"] ["message type"=CreatePartition] [collection=434328479435325441] [channel=by-dev-rootcoord-dml_2_434328479435325441v2] [2022/07/08 01:01:03.530 +08:00] [WARN] [flow_graph_filter_dm_node.go:118] ["invalid message type in filterDmNode"] ["message type"=CreatePartition] [collection=434328479448170498] [channel=by-dev-rootcoord-dml_11_434328479448170498v5] [2022/07/08 01:01:04.330 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434428513621377026] [collectionID=434328479553028101] [partitions="[434328479553028102,434328480012042241,434337916580003841,434360565899984898,434383215115370498,434405864134148097,434428513621377026]"] [2022/07/08 01:01:04.330 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434428513621377026] [2022/07/08 01:01:04.330 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434428513804615686] [segmentType=Growing] [2022/07/08 01:01:04.330 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434428513804615686] [segmentType=Growing] [2022/07/08 01:01:07.647 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 01:01:09.140 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 01:01:10.470 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 01:01:11.601 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 01:01:12.647 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 01:01:13.515 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 01:01:13.516 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434428513804615686] [state=3] [2022/07/08 01:01:13.516 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434428513804615686 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:1005 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 01:01:14.265 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 01:01:14.964 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 01:01:15.872 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 01:01:16.690 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:03.531 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434429457352884225] [segmentType=Growing] [2022/07/08 02:01:03.531 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434429457352884225] [segmentType=Growing] [2022/07/08 02:01:06.426 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:07.071 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:09.147 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:09.425 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:09.732 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:10.034 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:10.296 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:11.383 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:12.024 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:12.715 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:12.715 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434429457352884225] [state=3] [2022/07/08 02:01:12.715 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434429457352884225 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:656 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 02:01:13.363 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:14.075 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:14.377 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:14.641 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 02:01:14.892 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:03.730 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434430401123713029] [segmentType=Growing] [2022/07/08 03:01:03.731 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434430401123713029] [segmentType=Growing] [2022/07/08 03:01:06.450 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:07.061 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:07.621 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:08.520 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:09.145 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:09.731 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:10.298 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:10.924 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:10.924 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434430401123713029] [state=3] [2022/07/08 03:01:10.924 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434430401123713029 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:439 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 03:01:11.549 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:11.774 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:12.005 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:12.308 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:12.542 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:12.790 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 03:01:13.020 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:02.931 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434431344619028485] [segmentType=Growing] [2022/07/08 04:01:02.931 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434431344619028485] [segmentType=Growing] [2022/07/08 04:01:05.920 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:06.742 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:07.507 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:07.508 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434431344619028485] [state=3] [2022/07/08 04:01:07.508 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434431344619028485 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:281 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 04:01:08.292 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:08.847 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:09.386 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:09.935 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:10.497 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:10.716 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:10.961 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:11.206 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:01:13.670 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:02:14.000 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:02:14.769 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:02:15.558 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:02:16.223 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:02:16.592 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:02:17.373 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:02:17.374 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434431363309895681] [state=3] [2022/07/08 04:02:17.374 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434431363309895681 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:2381 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434428513804615686 compactionFrom:434431344619028485 compactionFrom:434430401123713029 compactionFrom:434429457352884225 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 04:02:17.374 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434428513804615686] [state=0] [2022/07/08 04:02:17.374 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434431344619028485] [state=0] [2022/07/08 04:02:17.374 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434430401123713029] [state=0] [2022/07/08 04:02:17.374 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434429457352884225] [state=0] [2022/07/08 04:02:17.375 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434428513804615686] [state=1] [2022/07/08 04:02:17.375 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434428513804615686] [2022/07/08 04:02:17.375 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434431344619028485] [state=1] [2022/07/08 04:02:17.375 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434431344619028485] [2022/07/08 04:02:17.376 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="offline_segments:<segmentID:434428513804615686 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:1005 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434431344619028485 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:281 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434430401123713029 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:439 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434429457352884225 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:656 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 04:02:17.376 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434430401123713029] [state=1] [2022/07/08 04:02:17.376 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434430401123713029] [2022/07/08 04:02:17.377 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434429457352884225] [state=1] [2022/07/08 04:02:17.377 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434429457352884225] [2022/07/08 04:02:17.748 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:03:14.219 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:03:14.635 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 04:04:13.588 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:04.531 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434432288507822081] [segmentType=Growing] [2022/07/08 05:01:04.531 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434432288507822081] [segmentType=Growing] [2022/07/08 05:01:06.549 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:07.470 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:08.244 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:08.895 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:09.578 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:09.579 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434432288507822081] [state=3] [2022/07/08 05:01:09.579 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434432288507822081 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:215 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 05:01:10.329 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:11.004 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:11.758 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:12.211 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:12.720 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:13.219 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:13.641 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:14.114 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 05:01:14.538 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:03.533 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434433232188473345] [segmentType=Growing] [2022/07/08 06:01:03.533 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434433232188473345] [segmentType=Growing] [2022/07/08 06:01:06.815 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:07.596 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:08.397 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:09.409 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:09.410 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434433232188473345] [state=3] [2022/07/08 06:01:09.410 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434433232188473345 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:218 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 06:01:10.288 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:11.522 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:11.918 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:12.390 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:12.850 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:13.380 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:13.863 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 06:01:14.297 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:03.130 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434434175892193281] [segmentType=Growing] [2022/07/08 07:01:03.130 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434434175892193281] [segmentType=Growing] [2022/07/08 07:01:06.225 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:06.963 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:06.964 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434434175892193281] [state=3] [2022/07/08 07:01:06.964 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434434175892193281 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:244 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 07:01:07.769 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:08.569 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:09.245 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:09.955 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:10.452 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:10.901 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:11.431 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:11.918 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:12.368 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 07:01:12.859 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:03.730 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434435119715450881] [segmentType=Growing] [2022/07/08 08:01:03.730 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434435119715450881] [segmentType=Growing] [2022/07/08 08:01:06.651 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:06.652 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434435119715450881] [state=3] [2022/07/08 08:01:06.652 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434435119715450881 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:356 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 08:01:07.438 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:08.174 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:08.971 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:09.810 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:10.575 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:11.131 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:11.609 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:12.076 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:12.546 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:13.109 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:01:13.626 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:03:13.886 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:03:14.843 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:03:15.720 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:03:16.567 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:03:16.567 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434435153715789825] [state=3] [2022/07/08 08:03:16.567 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434435153715789825 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:3058 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434431363309895681 compactionFrom:434432288507822081 compactionFrom:434433232188473345 compactionFrom:434434175892193281 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 08:03:16.567 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434431363309895681] [state=0] [2022/07/08 08:03:16.567 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434432288507822081] [state=0] [2022/07/08 08:03:16.567 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434433232188473345] [state=0] [2022/07/08 08:03:16.567 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434434175892193281] [state=0] [2022/07/08 08:03:16.568 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434431363309895681] [state=1] [2022/07/08 08:03:16.568 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434431363309895681] [2022/07/08 08:03:16.568 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="offline_segments:<segmentID:434431363309895681 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:2381 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434428513804615686 compactionFrom:434431344619028485 compactionFrom:434430401123713029 compactionFrom:434429457352884225 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434432288507822081 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:215 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434433232188473345 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:218 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434434175892193281 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:244 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 08:03:16.568 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434432288507822081] [state=1] [2022/07/08 08:03:16.568 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434432288507822081] [2022/07/08 08:03:16.569 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434433232188473345] [state=1] [2022/07/08 08:03:16.569 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434433232188473345] [2022/07/08 08:03:16.570 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434434175892193281] [state=1] [2022/07/08 08:03:16.570 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434434175892193281] [2022/07/08 08:03:17.446 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:03:17.858 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:03:18.327 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:04:14.640 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:04:15.550 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:04:15.975 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 08:04:16.407 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 09:01:02.931 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434436063198183429] [segmentType=Growing] [2022/07/08 09:01:02.931 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434436063198183429] [segmentType=Growing] [2022/07/08 09:01:05.985 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 09:01:06.667 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 09:01:07.570 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 09:01:08.395 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 09:01:09.176 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 09:01:09.900 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 09:01:09.900 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434436063198183429] [state=3] [2022/07/08 09:01:09.914 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434436063198183429 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:415 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 09:01:10.671 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:03.531 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434437007112929283] [segmentType=Growing] [2022/07/08 10:01:03.531 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434437007112929283] [segmentType=Growing] [2022/07/08 10:01:06.448 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:07.367 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:08.017 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:08.544 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:09.154 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:09.742 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:10.227 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:11.140 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:12.036 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:12.037 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434437007112929283] [state=3] [2022/07/08 10:01:12.037 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434437007112929283 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:547 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 10:01:12.856 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:13.819 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 10:01:14.596 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:03.930 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434437950883758082] [segmentType=Growing] [2022/07/08 11:01:03.931 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434437950883758082] [segmentType=Growing] [2022/07/08 11:01:06.773 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:07.439 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:08.002 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:08.523 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:09.051 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:09.554 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:10.574 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:11.411 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:11.412 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434437950883758082] [state=3] [2022/07/08 11:01:11.412 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434437950883758082 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:670 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 11:01:12.300 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:13.233 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:14.075 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:01:15.051 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:03:14.042 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:03:14.680 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:03:15.249 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:03:15.603 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:03:15.942 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:03:16.248 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:04:14.878 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:04:16.076 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:04:16.076 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434438000547201032] [state=3] [2022/07/08 11:04:16.077 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434438000547201032 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:5046 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434435153715789825 compactionFrom:434435119715450881 compactionFrom:434436063198183429 compactionFrom:434437007112929283 compactionFrom:434437950883758082 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 11:04:16.077 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434435153715789825] [state=0] [2022/07/08 11:04:16.077 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434435119715450881] [state=0] [2022/07/08 11:04:16.077 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434436063198183429] [state=0] [2022/07/08 11:04:16.077 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434437007112929283] [state=0] [2022/07/08 11:04:16.077 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434437950883758082] [state=0] [2022/07/08 11:04:16.078 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434435153715789825] [state=1] [2022/07/08 11:04:16.078 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434435153715789825] [2022/07/08 11:04:16.078 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434435119715450881] [state=1] [2022/07/08 11:04:16.078 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434435119715450881] [2022/07/08 11:04:16.079 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434436063198183429] [state=1] [2022/07/08 11:04:16.079 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434436063198183429] [2022/07/08 11:04:16.080 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434437007112929283] [state=1] [2022/07/08 11:04:16.080 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434437007112929283] [2022/07/08 11:04:16.081 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434437950883758082] [state=1] [2022/07/08 11:04:16.081 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434437950883758082] [2022/07/08 11:04:16.081 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="offline_segments:<segmentID:434435153715789825 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:3058 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434431363309895681 compactionFrom:434432288507822081 compactionFrom:434433232188473345 compactionFrom:434434175892193281 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434435119715450881 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:356 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434436063198183429 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:415 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434437007112929283 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:547 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434437950883758082 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:670 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 11:04:17.201 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:04:18.500 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:04:19.654 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:05:15.092 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 11:05:16.379 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:03.930 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434438894576205826] [segmentType=Growing] [2022/07/08 12:01:03.930 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434438894576205826] [segmentType=Growing] [2022/07/08 12:01:06.646 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:07.407 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:08.081 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:08.975 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:09.742 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:10.413 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:11.338 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:12.196 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:12.197 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434438894576205826] [state=3] [2022/07/08 12:01:12.206 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434438894576205826 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:601 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 12:01:13.120 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:13.958 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 12:01:14.816 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:03.930 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434439838320558083] [segmentType=Growing] [2022/07/08 13:01:03.930 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434439838320558083] [segmentType=Growing] [2022/07/08 13:01:06.611 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:07.512 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:08.055 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:08.591 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:09.096 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:09.618 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:10.190 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:11.092 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:11.093 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434439838320558083] [state=3] [2022/07/08 13:01:11.093 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434439838320558083 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:579 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 13:01:11.907 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:12.815 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:13.822 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 13:01:14.766 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:02.730 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434440781776814082] [segmentType=Growing] [2022/07/08 14:01:02.731 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434428513621377026] [segmentID=434440781776814082] [segmentType=Growing] [2022/07/08 14:01:05.671 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:06.582 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:07.246 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:07.838 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:08.371 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:08.936 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:09.559 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:10.583 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:11.679 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:12.607 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:13.445 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:14.285 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:01:14.286 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434440781776814082] [state=3] [2022/07/08 14:01:14.286 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434440781776814082 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:645 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 14:04:14.254 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:04:14.852 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:04:15.386 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:04:15.698 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:04:16.336 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:04:17.038 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:05:15.026 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:05:16.428 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:05:17.807 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:05:19.229 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:05:19.229 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434440847666708481] [state=3] [2022/07/08 14:05:19.229 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="online_segments:<segmentID:434440847666708481 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:6871 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434438000547201032 compactionFrom:434439838320558083 compactionFrom:434438894576205826 compactionFrom:434440781776814082 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 14:05:19.229 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434438000547201032] [state=0] [2022/07/08 14:05:19.229 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434439838320558083] [state=0] [2022/07/08 14:05:19.229 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434438894576205826] [state=0] [2022/07/08 14:05:19.229 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434440781776814082] [state=0] [2022/07/08 14:05:19.230 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434438000547201032] [state=1] [2022/07/08 14:05:19.230 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434438000547201032] [2022/07/08 14:05:19.230 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434439838320558083] [state=1] [2022/07/08 14:05:19.230 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434439838320558083] [2022/07/08 14:05:19.231 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_31_434328479553028101v1] [segment="offline_segments:<segmentID:434438000547201032 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:5046 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" compactionFrom:434435153715789825 compactionFrom:434435119715450881 compactionFrom:434436063198183429 compactionFrom:434437007112929283 compactionFrom:434437950883758082 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434439838320558083 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:579 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434438894576205826 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:601 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434440781776814082 collectionID:434328479553028101 partitionID:434428513621377026 num_rows:645 dmChannel:"by-dev-rootcoord-dml_31_434328479553028101v1" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/08 14:05:19.231 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434438894576205826] [state=1] [2022/07/08 14:05:19.231 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434438894576205826] [2022/07/08 14:05:19.232 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434440781776814082] [state=1] [2022/07/08 14:05:19.232 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434440781776814082] [2022/07/08 14:05:20.577 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:06:15.662 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:06:17.100 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:19:18.476 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:19:18.997 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:19:19.556 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:19:20.108 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:19:20.586 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:40:04.730 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:40:07.833 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:40:10.566 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:40:53.912 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/08 14:46:31.083 +08:00] [INFO] [impl.go:229] ["loadSegmentsTask Enqueue done"] [collectionID=434422255208628225] [segmentIDs="[434441135264890882]"] [nodeID=30] [2022/07/08 14:46:31.083 +08:00] [INFO] [task.go:507] ["LoadSegmentTask PreExecute start"] [msgID=0] [2022/07/08 14:46:31.083 +08:00] [INFO] [collection.go:329] ["create collection"] [collectionID=434422255208628225] [2022/07/08 14:46:31.083 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434422255208628226] [collectionID=434422255208628225] [partitions="[434422255208628226]"] [2022/07/08 14:46:31.083 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434422255208628226] [2022/07/08 14:46:31.083 +08:00] [INFO] [collection.go:87] ["queryNode collection info after add a partition"] [partitionID=434422285551796225] [collectionID=434422255208628225] [partitions="[434422255208628226,434422285551796225]"] [2022/07/08 14:46:31.083 +08:00] [INFO] [partition.go:114] ["create partition"] [partitionID=434422285551796225] [2022/07/08 14:46:31.083 +08:00] [INFO] [task.go:533] ["LoadSegmentTask PreExecute done"] [msgID=0] [2022/07/08 14:46:31.083 +08:00] [INFO] [task.go:539] ["LoadSegmentTask Execute start"] [msgID=0] [2022/07/08 14:46:31.083 +08:00] [INFO] [segment_loader.go:86] ["segmentLoader start loading..."] [collectionID=434422255208628225] [segmentNum=1] [segmentType=Sealed] [2022/07/08 14:46:31.083 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434422255208628225] [partitionID=434441135225831425] [segmentID=434441135264890882] [segmentType=Sealed] [2022/07/08 14:46:31.083 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434422255208628225] [partitionID=434441135225831425] [segmentID=434441135264890882] [segmentType=Sealed] [2022/07/08 14:46:31.312 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434441135264890882] [fieldID=101] [2022/07/08 14:46:31.312 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434422255208628225] [segment=434441135264890882] [fieldID=101] [2022/07/08 14:46:31.321 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=103] ["row count"=10721] [segmentID=434441135264890882] [2022/07/08 14:46:31.322 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=102] ["row count"=10721] [segmentID=434441135264890882] [2022/07/08 14:46:31.327 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=10721] [segmentID=434441135264890882] [2022/07/08 14:46:31.327 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=10721] [segmentID=434441135264890882] [2022/07/08 14:46:31.327 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=10721] [segmentID=434441135264890882] [2022/07/08 14:46:31.327 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434422255208628225] [segment=434441135264890882] [len(field)=5] [segmentType=Sealed] [2022/07/08 14:46:31.416 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434441135264890882] [2022/07/08 14:46:31.447 +08:00] [ERROR] [segment_loader.go:187] ["load segment failed, set segment to meta failed"] [collectionID=434422255208628225] [partitionID=434441135225831425] [segmentID=434441135264890882] ["loadSegmentRequest msgID"=0] [error="partition 434441135225831425 hasn't been loaded or has been released"] [stack="github.com/milvus-io/milvus/internal/querynode.(*segmentLoader).LoadSegment\n\t/opt/milvus/internal/querynode/segment_loader.go:187\ngithub.com/milvus-io/milvus/internal/querynode.(*loadSegmentsTask).Execute\n\t/opt/milvus/internal/querynode/task.go:540\ngithub.com/milvus-io/milvus/internal/querynode.(*taskScheduler).processTask\n\t/opt/milvus/internal/querynode/task_scheduler.go:109\ngithub.com/milvus-io/milvus/internal/querynode.(*taskScheduler).taskLoop\n\t/opt/milvus/internal/querynode/task_scheduler.go:126"] [2022/07/08 14:46:31.452 +08:00] [INFO] [segment.go:220] ["delete segment from memory"] [collectionID=434422255208628225] [partitionID=434441135225831425] [segmentID=434441135264890882] [segmentType=Sealed] [2022/07/08 14:46:31.457 +08:00] [WARN] [task.go:542] ["partition 434441135225831425 hasn't been loaded or has been released"] [2022/07/08 14:46:31.458 +08:00] [WARN] [task_scheduler.go:111] ["partition 434441135225831425 hasn't been loaded or has been released"] [2022/07/08 14:46:31.458 +08:00] [WARN] [impl.go:238] ["partition 434441135225831425 hasn't been loaded or has been released"] [2022/07/08 14:46:31.463 +08:00] [INFO] [impl.go:229] ["loadSegmentsTask Enqueue done"] [collectionID=434422255208628225] [segmentIDs="[434441135264890882]"] [nodeID=30] [2022/07/08 14:46:31.463 +08:00] [INFO] [task.go:507] ["LoadSegmentTask PreExecute start"] [msgID=0] [2022/07/08 14:46:31.463 +08:00] [INFO] [task.go:533] ["LoadSegmentTask PreExecute done"] [msgID=0] [2022/07/08 14:46:31.463 +08:00] [INFO] [task.go:539] ["LoadSegmentTask Execute start"] [msgID=0] [2022/07/08 14:46:31.463 +08:00] [INFO] [segment_loader.go:86] ["segmentLoader start loading..."] [collectionID=434422255208628225] [segmentNum=1] [segmentType=Sealed] [2022/07/08 14:46:31.463 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434422255208628225] [partitionID=434441135225831425] [segmentID=434441135264890882] [segmentType=Sealed] [2022/07/08 14:46:31.464 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434422255208628225] [partitionID=434441135225831425] [segmentID=434441135264890882] [segmentType=Sealed] [2022/07/08 14:46:31.646 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434441135264890882] [fieldID=101] [2022/07/08 14:46:31.646 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434422255208628225] [segment=434441135264890882] [fieldID=101] [2022/07/08 14:46:31.653 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=103] ["row count"=10721] [segmentID=434441135264890882] [2022/07/08 14:46:31.654 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=102] ["row count"=10721] [segmentID=434441135264890882] [2022/07/08 14:46:31.656 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=10721] [segmentID=434441135264890882] [2022/07/08 14:46:31.662 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=10721] [segmentID=434441135264890882] [2022/07/08 14:46:31.662 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=10721] [segmentID=434441135264890882] [2022/07/08 14:46:31.662 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434422255208628225] [segment=434441135264890882] [len(field)=5] [segmentType=Sealed] [2022/07/08 14:46:31.753 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434441135264890882] [2022/07/08 14:46:31.787 +08:00] [ERROR] [segment_loader.go:187] ["load segment failed, set segment to meta failed"] [collectionID=434422255208628225] [partitionID=434441135225831425] [segmentID=434441135264890882] ["loadSegmentRequest msgID"=0] [error="partition 434441135225831425 hasn't been loaded or has been released"] [stack="github.com/milvus-io/milvus/internal/querynode.(*segmentLoader).LoadSegment\n\t/opt/milvus/internal/querynode/segment_loader.go:187\ngithub.com/milvus-io/milvus/internal/querynode.(*loadSegmentsTask).Execute\n\t/opt/milvus/internal/querynode/task.go:540\ngithub.com/milvus-io/milvus/internal/querynode.(*taskScheduler).processTask\n\t/opt/milvus/internal/querynode/task_scheduler.go:109\ngithub.com/milvus-io/milvus/internal/querynode.(*taskScheduler).taskLoop\n\t/opt/milvus/internal/querynode/task_scheduler.go:126"]

Jul 13 '22 07:07 yhmo

partial log of query node 005:

[2022/07/09 11:04:22.420 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434457768747597828] [state=0] [2022/07/09 11:04:22.420 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434458712439783426] [state=0] [2022/07/09 11:04:22.420 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434459656367636482] [state=0] [2022/07/09 11:04:22.420 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434460650195124225]"] [Scope=Streaming] [2022/07/09 11:04:22.420 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434457818974388225] [state=1] [2022/07/09 11:04:22.421 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434457818974388225] [2022/07/09 11:04:22.421 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434457818974388225,434457768747597828,434458712439783426,434459656367636482]"] [Scope=Streaming] [2022/07/09 11:04:22.421 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434457768747597828] [state=1] [2022/07/09 11:04:22.421 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434457768747597828] [2022/07/09 11:04:22.422 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434458712439783426] [state=1] [2022/07/09 11:04:22.422 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="offline_segments:<segmentID:434457818974388225 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:2965 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" compactionFrom:434454059645665281 compactionFrom:434455881376333825 compactionFrom:434454937671041029 compactionFrom:434456825029197826 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434457768747597828 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:331 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434458712439783426 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:393 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434459656367636482 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:523 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 11:04:22.422 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434458712439783426] [2022/07/09 11:04:22.422 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434459656367636482] [state=1] [2022/07/09 11:04:22.422 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434459656367636482] [2022/07/09 11:05:15.697 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 11:05:16.181 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 11:05:16.947 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 12:01:03.330 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434461543726055427] [segmentType=Growing] [2022/07/09 12:01:03.330 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434461543726055427] [segmentType=Growing] [2022/07/09 12:01:03.330 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434461543726055429] [segmentType=Growing] [2022/07/09 12:01:03.330 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434461543726055429] [segmentType=Growing] [2022/07/09 12:01:06.564 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 12:01:07.408 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 12:01:07.409 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434461543726055427] [state=3] [2022/07/09 12:01:07.410 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_32_434328479553028101v2] [segment="online_segments:<segmentID:434461543726055427 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:646 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 12:01:08.117 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 12:01:08.866 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 12:01:09.593 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 12:01:10.471 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 12:01:11.430 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 12:01:12.413 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 12:01:12.414 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434461543726055429] [state=3] [2022/07/09 12:01:12.414 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="online_segments:<segmentID:434461543726055429 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:677 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 12:01:12.414 +08:00] [INFO] [partition.go:100] ["remove a segment from replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434461543726055429] [segmentType=Growing] [2022/07/09 12:01:12.415 +08:00] [INFO] [segment.go:220] ["delete segment from memory"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434461543726055429] [segmentType=Growing] [2022/07/09 12:01:12.415 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434461543726055429]"] [Scope=Streaming] [2022/07/09 12:01:13.244 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 12:01:14.127 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:03.129 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434462487392026630] [segmentType=Growing] [2022/07/09 13:01:03.131 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434462487392026630] [segmentType=Growing] [2022/07/09 13:01:03.131 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434462487392026625] [segmentType=Growing] [2022/07/09 13:01:03.131 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434462487392026625] [segmentType=Growing] [2022/07/09 13:01:06.592 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:07.243 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:07.913 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:08.514 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:09.143 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:09.722 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:10.272 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:11.212 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:11.213 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434462487392026630] [state=3] [2022/07/09 13:01:11.213 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="online_segments:<segmentID:434462487392026630 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:657 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 13:01:11.213 +08:00] [INFO] [partition.go:100] ["remove a segment from replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434462487392026630] [segmentType=Growing] [2022/07/09 13:01:11.214 +08:00] [INFO] [segment.go:220] ["delete segment from memory"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434462487392026630] [segmentType=Growing] [2022/07/09 13:01:11.215 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434462487392026630]"] [Scope=Streaming] [2022/07/09 13:01:12.077 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:12.884 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:12.884 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434462487392026625] [state=3] [2022/07/09 13:01:12.885 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_32_434328479553028101v2] [segment="online_segments:<segmentID:434462487392026625 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:605 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 13:01:13.795 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:14.641 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:15.218 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:15.770 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:16.338 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:16.935 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:17.452 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:01:17.994 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:02:15.134 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:03:15.152 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:03:16.507 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:03:16.508 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434462505768058884] [state=3] [2022/07/09 13:03:16.509 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="online_segments:<segmentID:434462505768058884 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:6154 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" compactionFrom:434460650195124225 compactionFrom:434460599915905026 compactionFrom:434462487392026630 compactionFrom:434461543726055429 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 13:03:16.509 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434460650195124225] [state=0] [2022/07/09 13:03:16.509 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434460599915905026] [state=0] [2022/07/09 13:03:16.509 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434462487392026630] [state=0] [2022/07/09 13:03:16.509 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434461543726055429] [state=0] [2022/07/09 13:03:16.509 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434462505768058884]"] [Scope=Streaming] [2022/07/09 13:03:16.509 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434460650195124225] [state=1] [2022/07/09 13:03:16.509 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434460650195124225] [2022/07/09 13:03:16.510 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434460650195124225,434460599915905026,434462487392026630,434461543726055429]"] [Scope=Streaming] [2022/07/09 13:03:16.510 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434460599915905026] [state=1] [2022/07/09 13:03:16.510 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434460599915905026] [2022/07/09 13:03:16.510 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434462487392026630] [state=1] [2022/07/09 13:03:16.510 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434462487392026630] [2022/07/09 13:03:16.510 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="offline_segments:<segmentID:434460650195124225 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:4212 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" compactionFrom:434457818974388225 compactionFrom:434457768747597828 compactionFrom:434458712439783426 compactionFrom:434459656367636482 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434460599915905026 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:608 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434462487392026630 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:657 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434461543726055429 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:677 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 13:03:16.511 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434461543726055429] [state=1] [2022/07/09 13:03:16.511 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434461543726055429] [2022/07/09 13:03:18.167 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:04:14.460 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:04:15.291 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:04:16.212 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 13:04:17.028 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 14:01:03.131 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434463431083950082] [segmentType=Growing] [2022/07/09 14:01:03.131 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434463431083950082] [segmentType=Growing] [2022/07/09 14:01:03.131 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434463431083950083] [segmentType=Growing] [2022/07/09 14:01:03.131 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434463431083950083] [segmentType=Growing] [2022/07/09 14:01:06.688 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 14:01:06.689 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434462537225601029] [state=3] [2022/07/09 14:01:06.689 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_32_434328479553028101v2] [segment="online_segments:<segmentID:434462537225601029 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:6079 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" compactionFrom:434460649788801028 compactionFrom:434462487392026625 compactionFrom:434460599915905029 compactionFrom:434461543726055427 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 14:01:06.689 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434460649788801028] [state=0] [2022/07/09 14:01:06.689 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434462487392026625] [state=0] [2022/07/09 14:01:06.689 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434460599915905029] [state=0] [2022/07/09 14:01:06.689 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434461543726055427] [state=0] [2022/07/09 14:01:06.690 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434460649788801028] [state=1] [2022/07/09 14:01:06.690 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434460649788801028] [2022/07/09 14:01:06.690 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434462487392026625] [state=1] [2022/07/09 14:01:06.690 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434462487392026625] [2022/07/09 14:01:06.691 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_32_434328479553028101v2] [segment="offline_segments:<segmentID:434460649788801028 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:4194 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" compactionFrom:434457819288961025 compactionFrom:434457768747597826 compactionFrom:434458712426414082 compactionFrom:434459656354529282 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434462487392026625 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:605 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434460599915905029 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:634 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434461543726055427 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:646 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 14:01:06.691 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434460599915905029] [state=1] [2022/07/09 14:01:06.691 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434460599915905029] [2022/07/09 14:01:06.692 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434461543726055427] [state=1] [2022/07/09 14:01:06.692 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434461543726055427] [2022/07/09 14:01:07.744 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 14:01:09.195 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 14:01:10.100 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 14:01:10.101 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434463431083950083] [state=3] [2022/07/09 14:01:10.101 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="online_segments:<segmentID:434463431083950083 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:702 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 14:01:10.101 +08:00] [INFO] [partition.go:100] ["remove a segment from replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434463431083950083] [segmentType=Growing] [2022/07/09 14:01:10.102 +08:00] [INFO] [segment.go:220] ["delete segment from memory"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434463431083950083] [segmentType=Growing] [2022/07/09 14:01:10.102 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434463431083950083]"] [Scope=Streaming] [2022/07/09 14:01:10.984 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 14:01:10.984 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434463431083950082] [state=3] [2022/07/09 14:01:10.985 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_32_434328479553028101v2] [segment="online_segments:<segmentID:434463431083950082 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:680 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 14:01:12.079 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 14:01:13.113 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 14:01:14.069 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:03.530 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434464374907207681] [segmentType=Growing] [2022/07/09 15:01:03.530 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434464374907207681] [segmentType=Growing] [2022/07/09 15:01:03.530 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434464374920577026] [segmentType=Growing] [2022/07/09 15:01:03.531 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434464374920577026] [segmentType=Growing] [2022/07/09 15:01:06.525 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:07.156 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:07.841 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:08.474 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:09.044 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:09.932 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:09.933 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434464374907207681] [state=3] [2022/07/09 15:01:09.933 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_32_434328479553028101v2] [segment="online_segments:<segmentID:434464374907207681 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:701 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 15:01:10.497 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:11.607 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:12.737 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:12.738 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434464374920577026] [state=3] [2022/07/09 15:01:12.738 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="online_segments:<segmentID:434464374920577026 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:666 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 15:01:12.738 +08:00] [INFO] [partition.go:100] ["remove a segment from replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434464374920577026] [segmentType=Growing] [2022/07/09 15:01:12.739 +08:00] [INFO] [segment.go:220] ["delete segment from memory"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434464374920577026] [segmentType=Growing] [2022/07/09 15:01:12.739 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434464374920577026]"] [Scope=Streaming] [2022/07/09 15:01:13.631 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:14.720 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:01:15.866 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:04:15.607 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:04:16.287 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:04:16.590 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:04:16.989 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:04:17.654 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:04:18.004 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 15:05:16.427 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 16:01:03.731 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434465318664929285] [segmentType=Growing] [2022/07/09 16:01:03.731 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434465318664929285] [segmentType=Growing] [2022/07/09 16:01:03.733 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434465318664929284] [segmentType=Growing] [2022/07/09 16:01:03.733 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434465318664929284] [segmentType=Growing] [2022/07/09 16:01:06.953 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 16:01:08.028 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 16:01:09.075 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 16:01:09.964 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 16:01:11.087 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 16:01:11.088 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434465318664929284] [state=3] [2022/07/09 16:01:11.088 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="online_segments:<segmentID:434465318664929284 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:686 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 16:01:11.089 +08:00] [INFO] [partition.go:100] ["remove a segment from replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434465318664929284] [segmentType=Growing] [2022/07/09 16:01:11.089 +08:00] [INFO] [segment.go:220] ["delete segment from memory"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434465318664929284] [segmentType=Growing] [2022/07/09 16:01:11.089 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434465318664929284]"] [Scope=Streaming] [2022/07/09 16:01:11.997 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 16:01:11.998 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434465318664929285] [state=3] [2022/07/09 16:01:11.998 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_32_434328479553028101v2] [segment="online_segments:<segmentID:434465318664929285 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:682 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 16:01:12.984 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 16:01:13.626 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 16:01:14.251 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:01:03.331 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434466262265364481] [segmentType=Growing] [2022/07/09 17:01:03.331 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434466262265364481] [segmentType=Growing] [2022/07/09 17:01:03.332 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434466262265364484] [segmentType=Growing] [2022/07/09 17:01:03.332 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434466262265364484] [segmentType=Growing] [2022/07/09 17:01:05.989 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:01:06.790 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:01:07.819 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:01:07.819 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434466262265364481] [state=3] [2022/07/09 17:01:07.819 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_32_434328479553028101v2] [segment="online_segments:<segmentID:434466262265364481 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:699 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 17:01:08.607 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:01:09.329 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:01:10.422 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:01:10.423 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434466262265364484] [state=3] [2022/07/09 17:01:10.423 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="online_segments:<segmentID:434466262265364484 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:737 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 17:01:10.423 +08:00] [INFO] [partition.go:100] ["remove a segment from replica"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434466262265364484] [segmentType=Growing] [2022/07/09 17:01:10.424 +08:00] [INFO] [segment.go:220] ["delete segment from memory"] [collectionID=434328479553028101] [partitionID=434451162731905026] [segmentID=434466262265364484] [segmentType=Growing] [2022/07/09 17:01:10.424 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434466262265364484]"] [Scope=Streaming] [2022/07/09 17:01:11.488 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:01:12.588 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:01:13.550 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:01:14.586 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:02:14.474 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:02:15.248 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:02:15.881 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:02:16.450 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:02:17.112 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:02:17.734 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:03:14.820 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:03:15.911 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:04:15.621 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:04:16.830 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:04:18.628 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:04:18.629 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434466296475156481] [state=3] [2022/07/09 17:04:18.629 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="online_segments:<segmentID:434466296475156481 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:8945 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" compactionFrom:434462505768058884 compactionFrom:434464374920577026 compactionFrom:434465318664929284 compactionFrom:434463431083950083 compactionFrom:434466262265364484 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 17:04:18.629 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434462505768058884] [state=0] [2022/07/09 17:04:18.629 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434464374920577026] [state=0] [2022/07/09 17:04:18.629 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434465318664929284] [state=0] [2022/07/09 17:04:18.629 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434463431083950083] [state=0] [2022/07/09 17:04:18.629 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434466262265364484] [state=0] [2022/07/09 17:04:18.630 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434466296475156481]"] [Scope=Streaming] [2022/07/09 17:04:18.630 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434462505768058884] [state=1] [2022/07/09 17:04:18.630 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434462505768058884] [2022/07/09 17:04:18.630 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434328479553028101] [segmentIDs="[434462505768058884,434464374920577026,434465318664929284,434463431083950083,434466262265364484]"] [Scope=Streaming] [2022/07/09 17:04:18.630 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434464374920577026] [state=1] [2022/07/09 17:04:18.630 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434464374920577026] [2022/07/09 17:04:18.631 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_33_434328479553028101v3] [segment="offline_segments:<segmentID:434462505768058884 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:6154 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" compactionFrom:434460650195124225 compactionFrom:434460599915905026 compactionFrom:434462487392026630 compactionFrom:434461543726055429 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434464374920577026 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:666 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434465318664929284 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:686 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434463431083950083 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:702 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434466262265364484 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:737 dmChannel:"by-dev-rootcoord-dml_33_434328479553028101v3" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 17:04:18.632 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434465318664929284] [state=1] [2022/07/09 17:04:18.632 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434465318664929284] [2022/07/09 17:04:18.633 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434463431083950083] [state=1] [2022/07/09 17:04:18.633 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434463431083950083] [2022/07/09 17:04:18.634 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434466262265364484] [state=1] [2022/07/09 17:04:18.634 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434466262265364484] [2022/07/09 17:04:20.599 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:04:20.600 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=29] [segmentID=434466312098938886] [state=3] [2022/07/09 17:04:20.600 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434462537225601029] [state=1] [2022/07/09 17:04:20.600 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434463431083950082] [state=1] [2022/07/09 17:04:20.600 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_32_434328479553028101v2] [segment="online_segments:<segmentID:434466312098938886 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:8841 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" compactionFrom:434462537225601029 compactionFrom:434463431083950082 compactionFrom:434465318664929285 compactionFrom:434466262265364481 compactionFrom:434464374907207681 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 17:04:20.600 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434465318664929285] [state=1] [2022/07/09 17:04:20.600 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434462537225601029] [state=0] [2022/07/09 17:04:20.600 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434462537225601029] [2022/07/09 17:04:20.600 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434463431083950082] [state=0] [2022/07/09 17:04:20.601 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434463431083950082] [2022/07/09 17:04:20.601 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434465318664929285] [state=0] [2022/07/09 17:04:20.601 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434465318664929285] [2022/07/09 17:04:20.601 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434466262265364481] [state=0] [2022/07/09 17:04:20.601 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434464374907207681] [state=0] [2022/07/09 17:04:20.601 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434466262265364481] [state=1] [2022/07/09 17:04:20.601 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434466262265364481] [2022/07/09 17:04:20.601 +08:00] [INFO] [shard_cluster.go:358] ["ShardCluster remove segment"] [nodeID=29] [segmentID=434464374907207681] [state=1] [2022/07/09 17:04:20.602 +08:00] [WARN] [shard_cluster.go:365] ["ShardCluster removeSegment does not belong to it"] [nodeID=29] [segmentID=434464374907207681] [2022/07/09 17:04:20.603 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_32_434328479553028101v2] [segment="offline_segments:<segmentID:434462537225601029 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:6079 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" compactionFrom:434460649788801028 compactionFrom:434462487392026625 compactionFrom:434460599915905029 compactionFrom:434461543726055427 segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434463431083950082 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:680 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434465318664929285 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:682 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434466262265364481 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:699 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > offline_segments:<segmentID:434464374907207681 collectionID:434328479553028101 partitionID:434451162731905026 num_rows:701 dmChannel:"by-dev-rootcoord-dml_32_434328479553028101v2" segment_state:Sealed replica_ids:434327753245589546 node_ids:29 > "] [2022/07/09 17:07:41.730 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434422255208628225] [partitionID=434450202655457281] [segmentID=434466366769332225] [segmentType=Growing] [2022/07/09 17:07:41.731 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434422255208628225] [partitionID=434450202655457281] [segmentID=434466366769332225] [segmentType=Growing] [2022/07/09 17:07:44.433 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:07:45.295 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:07:46.009 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:07:46.687 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:07:47.465 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:07:48.601 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:10:32.123 +08:00] [INFO] [impl.go:229] ["loadSegmentsTask Enqueue done"] [collectionID=434422255208628225] [segmentIDs="[434450202813005825]"] [nodeID=31] [2022/07/09 17:10:32.123 +08:00] [INFO] [task.go:507] ["LoadSegmentTask PreExecute start"] [msgID=0] [2022/07/09 17:10:32.123 +08:00] [INFO] [task.go:533] ["LoadSegmentTask PreExecute done"] [msgID=0] [2022/07/09 17:10:32.123 +08:00] [INFO] [task.go:539] ["LoadSegmentTask Execute start"] [msgID=0] [2022/07/09 17:10:32.123 +08:00] [INFO] [segment_loader.go:86] ["segmentLoader start loading..."] [collectionID=434422255208628225] [segmentNum=1] [segmentType=Sealed] [2022/07/09 17:10:32.124 +08:00] [INFO] [segment.go:186] ["create segment"] [collectionID=434422255208628225] [partitionID=434450202655457281] [segmentID=434450202813005825] [segmentType=Sealed] [2022/07/09 17:10:32.124 +08:00] [INFO] [segment_loader.go:206] ["start loading segment data into memory"] [collectionID=434422255208628225] [partitionID=434450202655457281] [segmentID=434450202813005825] [segmentType=Sealed] [2022/07/09 17:10:33.105 +08:00] [INFO] [segment.go:796] ["updateSegmentIndex done"] [segmentID=434450202813005825] [fieldID=101] [2022/07/09 17:10:33.105 +08:00] [INFO] [segment_loader.go:433] ["load field binlogs done for sealed segment with index"] [collection=434422255208628225] [segment=434450202813005825] [fieldID=101] [2022/07/09 17:10:33.208 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=102] ["row count"=128563] [segmentID=434450202813005825] [2022/07/09 17:10:33.250 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=103] ["row count"=128563] [segmentID=434450202813005825] [2022/07/09 17:10:33.254 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=0] ["row count"=128563] [segmentID=434450202813005825] [2022/07/09 17:10:33.257 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=1] ["row count"=128563] [segmentID=434450202813005825] [2022/07/09 17:10:33.320 +08:00] [INFO] [segment.go:700] ["load field done"] [fieldID=100] ["row count"=128563] [segmentID=434450202813005825] [2022/07/09 17:10:33.320 +08:00] [INFO] [segment_loader.go:361] ["load field binlogs done for sealed segment"] [collection=434422255208628225] [segment=434450202813005825] [len(field)=5] [segmentType=Sealed] [2022/07/09 17:10:34.495 +08:00] [INFO] [segment_loader.go:617] ["there are no delta logs saved with segment, skip loading delete record"] [segmentID=434450202813005825] [2022/07/09 17:10:34.644 +08:00] [INFO] [partition.go:74] ["add a segment to replica"] [collectionID=434422255208628225] [partitionID=434450202655457281] [segmentID=434450202813005825] [segmentType=Sealed] [2022/07/09 17:10:34.644 +08:00] [INFO] [segment_loader.go:633] ["from dml check point load delete"] [position="channel_name:"by-dev-rootcoord-dml_0_434422255208628225v0" msgID:"\010\345\215\022\020\275\313\001\030\000 \000" msgGroup:"by-dev-dataNode-22-434422255208628225" timestamp:434466366756225033 "] ["msg id"=COWNEhC9ywEYACAA] [2022/07/09 17:10:34.654 +08:00] [INFO] [mq_msgstream.go:174] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_0] [subname=querynode-31-434422255208628225] [2022/07/09 17:10:34.669 +08:00] [INFO] [mq_msgstream.go:552] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_0] [MessageID={}] [2022/07/09 17:10:34.783 +08:00] [INFO] [mq_msgstream.go:558] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_0] [2022/07/09 17:10:34.784 +08:00] [INFO] [segment_loader.go:671] ["start read delta msg from seek position to last position"] ["Collection ID"=434422255208628225] [channel=by-dev-rootcoord-dml_0] [2022/07/09 17:10:34.907 +08:00] [INFO] [segment_loader.go:724] ["All data has been read, there is no more data"] ["Collection ID"=434422255208628225] [channel=by-dev-rootcoord-dml_0] ["msg id"=COWNEhC9ywEYACAA] [2022/07/09 17:10:34.907 +08:00] [INFO] [segment_loader.go:742] ["from dml check point load done"] ["msg id"=COWNEhC9ywEYACAA] [2022/07/09 17:10:34.913 +08:00] [INFO] [task.go:558] ["LoadSegmentTask Execute done"] [msgID=0] [2022/07/09 17:10:34.913 +08:00] [INFO] [impl.go:241] ["loadSegmentsTask WaitToFinish done"] [collectionID=434422255208628225] [segmentIDs="[434450202813005825]"] [nodeID=31] [2022/07/09 17:10:34.917 +08:00] [INFO] [query_node.go:349] ["get SealedSegmentsChangeInfo from etcd"] [infoID=0] [2022/07/09 17:10:34.918 +08:00] [INFO] [shard_cluster.go:252] ["ShardCluster update segment"] [nodeID=31] [segmentID=434450202813005825] [state=3] [2022/07/09 17:10:34.918 +08:00] [INFO] [partition.go:100] ["remove a segment from replica"] [collectionID=434422255208628225] [partitionID=434450202655457281] [segmentID=434450202813005825] [segmentType=Growing] [2022/07/09 17:10:34.934 +08:00] [INFO] [segment.go:220] ["delete segment from memory"] [collectionID=434422255208628225] [partitionID=434450202655457281] [segmentID=434450202813005825] [segmentType=Growing] [2022/07/09 17:10:34.934 +08:00] [INFO] [impl.go:377] ["release segments done"] [collectionID=434422255208628225] [segmentIDs="[434450202813005825]"] [Scope=Streaming] [2022/07/09 17:10:34.934 +08:00] [INFO] [shard_cluster_service.go:156] ["successfully handoff"] [channel=by-dev-rootcoord-dml_0_434422255208628225v0] [segment="online_segments:<segmentID:434450202813005825 collectionID:434422255208628225 partitionID:434450202655457281 num_rows:128563 dmChannel:"by-dev-rootcoord-dml_0_434422255208628225v0" segment_state:Sealed replica_ids:434421960400175223 node_ids:31 > "] [2022/07/09 17:11:41.603 +08:00] [WARN] [grpclog.go:46] ["[core]grpc: addrConn.createTransport failed to connect to {10.97.103.31:21123 10.97.103.31:21123 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp 10.97.103.31:21123: connect: connection refused""] [2022/07/09 17:11:41.603 +08:00] [ERROR] [client.go:233] ["ClientBase Call grpc first call get error"] [role=querynode] [error="err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.97.103.31:21123: connect: connection refused"\n, /opt/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/opt/milvus/internal/util/grpcclient/client.go:232 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n/opt/milvus/internal/distributed/querynode/client/client.go:227 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/opt/milvus/internal/querynode/shard_cluster.go:781 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n/usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n\t/opt/milvus/internal/util/grpcclient/client.go:233\ngithub.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n\t/opt/milvus/internal/distributed/querynode/client/client.go:227\ngithub.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n\t/opt/milvus/internal/querynode/shard_cluster.go:781"] [2022/07/09 17:11:41.603 +08:00] [ERROR] [shard_cluster.go:798] ["Search 26 failed, reason err err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.97.103.31:21123: connect: connection refused"\n, /opt/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/opt/milvus/internal/util/grpcclient/client.go:232 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase).Call\n/opt/milvus/internal/distributed/querynode/client/client.go:227 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Search\n/opt/milvus/internal/querynode/shard_cluster.go:781 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search.func2\n/usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Search\n\t/opt/milvus/internal/querynode/shard_cluster.go:798\ngithub.com/milvus-io/milvus/internal/querynode.(*QueryNode).searchWithDmlChannel\n\t/opt/milvus/internal/querynode/impl.go:615\ngithub.com/milvus-io/milvus/internal/querynode.(*QueryNode).Search.func1\n\t/opt/milvus/internal/querynode/impl.go:463\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/gopath/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57"]

Jul 13 '22 07:07 yhmo

Milvus version: master(commitID: c1a47c89416f96de450a512bd349a6f4495edbb6)
Deployment mode(standalone or cluster): cluster
SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.0.1
OS(Ubuntu or CentOS): CentOS
CPU/Memory:
GPU:
Others:

datanode count: 2 (32cores/192GB)
indexnode count: 2 (32cores/192GB)
querynode count: 7 (32cores/256GB)

There are 9 collections in the cluster. The Largest scale collection has about 10 million rows, other collections range in size from a few hundred thousand to a few million. The entire cluster data volume is more than 20 million. The qps of every collection is around a few dozen.

Some collections are inserting a batch of data(thousands of rows) every hour, while others are continuously inserting data(qps < 10).

Jul 13 '22 08:07 BUPTAnderson

@yhmo checked all .out files(which I guess is the err output), all query node panicked due to Segment Violation:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x7fb28c000018 pc=0x7fbeebc5644a]

runtime stack:
runtime.throw(0x2e6f15b, 0x2a)
	/usr/local/go/src/runtime/panic.go:1117 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:718 +0x2e5

goroutine 165543018 [syscall]:
runtime.cgocall(0x20c2d9d, 0xc004af3788, 0xc001a99301)
	/usr/local/go/src/runtime/cgocall.go:154 +0x5b fp=0xc004af3758 sp=0xc004af3720 pc=0x5eab1b
github.com/milvus-io/milvus/internal/util/memutil._Cfunc_PurgeMemory(0xce964666, 0xc0057b49dc, 0xc000000000, 0x0)
	_cgo_gotypes.go:76 +0x51 fp=0xc004af3788 sp=0xc004af3758 pc=0x1f93831
github.com/milvus-io/milvus/internal/util/memutil.PurgeMemory(0xce964666, 0x5ffc00, 0x0, 0x0)
	/opt/milvus/internal/util/memutil/purge_memory.go:30 +0x85 fp=0xc004af3820 sp=0xc004af3788 pc=0x1f939a5
github.com/milvus-io/milvus/internal/querynode.deleteSearchResultDataBlobs(0x7fb8c4db0510)
	/opt/milvus/internal/querynode/reduce.go:147 +0x99 fp=0xc004af3938 sp=0xc004af3820 pc=0x1ffe7b9
github.com/milvus-io/milvus/internal/querynode.(*searchTask).reduceResults(0xc000c61180, 0xc0057c60e0, 0xc008a02580, 0x7, 0x7, 0x0, 0x0)
	/opt/milvus/internal/querynode/task_search.go:278 +0x2bb fp=0xc004af3c40 sp=0xc004af3938 pc=0x1fe945b
github.com/milvus-io/milvus/internal/querynode.(*searchTask).searchOnHistorical(0xc000c61180, 0x0, 0x0)
	/opt/milvus/internal/querynode/task_search.go:159 +0x34f fp=0xc004af3ec8 sp=0xc004af3c40 pc=0x1fe82cf
github.com/milvus-io/milvus/internal/querynode.(*searchTask).Execute(0xc000c61180, 0x31b71d0, 0xc0005c5f00, 0x0, 0x0)
	/opt/milvus/internal/querynode/task_search.go:166 +0x3c fp=0xc004af3f10 sp=0xc004af3ec8 pc=0x1fe871c
github.com/milvus-io/milvus/internal/querynode.(*taskScheduler).processReadTask(0xc0005c2b80, 0x31e10d8, 0xc000c61180)
	/opt/milvus/internal/querynode/task_scheduler.go:324 +0xdb fp=0xc004af3f80 sp=0xc004af3f10 pc=0x1fe651b
github.com/milvus-io/milvus/internal/querynode.(*taskScheduler).executeReadTasks.func1(0xc000b84800, 0xc0005c2b80, 0x31e10d8, 0xc000c61180)
	/opt/milvus/internal/querynode/task_scheduler.go:277 +0x6b fp=0xc004af3fc0 sp=0xc004af3f80 pc=0x200c08b
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc004af3fc8 sp=0xc004af3fc0 pc=0x65a6c1
created by github.com/milvus-io/milvus/internal/querynode.(*taskScheduler).executeReadTasks
	/opt/milvus/internal/querynode/task_scheduler.go:275 +0x2b6

Jul 13 '22 09:07 congqixia

@BUPTAnderson could you please provide the actual commit you compile with? Need to verify whether this problem still persist. Thanks in advance!

Jul 13 '22 09:07 congqixia

@congqixia master(commitID: c1a47c89416f96de450a512bd349a6f4495edbb6) (2022-07-07) Please confirm if the issue exists in branch-2.1.0, it would help me a lot.

Jul 13 '22 09:07 BUPTAnderson

/assign @bigsheeper

Jul 13 '22 09:07 congqixia

Hi @BUPTAnderson , https://github.com/milvus-io/milvus/pull/18311 has been merged, this pr added a exception safe protection. Would you please task a test with the latest version of master branch? Thank you very much.

Jul 20 '22 01:07 bigsheeper

/unassign /assign @BUPTAnderson

Jul 20 '22 01:07 bigsheeper

@bigsheeper Ok, I will verify it in the next two days.

Jul 20 '22 01:07 BUPTAnderson

@bigsheeper I updated the cluster last week on branch-2.1.0(commitID: 4844219e). I found that #18271 and #18349 only merged to master branch. I merged #18271 to branch-2.1.0 and deployed(pr #18349 has so many changes that I didn't merge to branch-2.1.0).
Unfortunately, I found that some queryNode still crashed after a periods of time, just less often, but not completely gone.

Jul 25 '22 09:07 BUPTAnderson

@BUPTAnderson I am so sorry for the late reply. As you can see, Milvus is now using Jemalloc as the memory allocator. QueryNode should not crash by _Cfunc_PurgeMemory anymore. You can try to deploy with the newest version.

Aug 11 '22 09:08 bigsheeper

@bigsheeper Will this feature be merged to branch-2.1.0 later? It seems to be better to deploy the release-2.1.0 in a production environment.

Aug 11 '22 10:08 BUPTAnderson

@bigsheeper Will this feature be merged to branch-2.1.0 later? It seems to be better to deploy the release-2.1.0 in a production environment.

Hi @BUPTAnderson , maybe there are some risks with Jemalloc. We need to take a long time to make sure it's stable. So this feature would be enable in 2.1.2 or later.

Aug 12 '22 08:08 bigsheeper

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

Sep 11 '22 08:09 stale[bot]

milvus milvus copied to clipboard

[Bug]: query node crashed and search timeout

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

milvus
milvus copied to clipboard