milvus
milvus copied to clipboard
Published
20 hours ago •
milvus-io
milvus-io
[Bug]: [laion1b-test] search failed: Assert "0 <= offsets[index] && offsets[index] < segment->get_row_count()"
Is there an existing issue for this?
- [X] I have searched the existing issues
Environment
- Milvus version: cardinal-milvus-io-2.3-66c8465-20240103
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
-
The collection
laion_coll_4has a partition-key field and 128 num_partitions. Collection has 50m-768dim data, DISKANN index -
concurrent insert + delete + flush + search + query
-
some search requests failed with error:
[2024-01-05 16:54:30,793 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=65535, message=failed to search/query delegator 746 for channel laion1b-test-2-rootcoord-dml_0_446771525888510903v0: fail to Search, QueryNode ID=746, reason=worker(748) query failed: Assert "0 <= offsets[index] && offsets[index] < segment->get_row_count()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/Reduce.cpp:97
=> invalid offset 911, segment 446771525947819298 with rows num 4, data or index corruption: service internal error: UnknownError)>, <Time:{'RPC start': '2024-01-05 16:52:05.834443', 'RPC error': '2024-01-05 16:54:30.793345'}> (decorators.py:128)
[2024-01-05 16:54:30,794 - ERROR - fouram]: (api_response) : <MilvusException: (code=65535, message=failed to search/query delegator 746 for channel laion1b-test-2-rootcoord-dml_0_446771525888510903v0: fail to Search, QueryNode ID=746, reason=worker(748) query failed: Assert "0 <= offsets[index] && offsets[index] < segment->get_row_count()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/Reduce.cpp:97
=> invalid offset 911, segment 446771525947819298 with rows num 4, data or index corruption: service internal error: UnknownError)> (api_request.py:54)
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
pods:
laion1b-test-2-milvus-datanode-dc8c8b9db-gbmjr 1/1 Running 0 5d15h
laion1b-test-2-milvus-datanode-dc8c8b9db-jkbn6 1/1 Running 1 (3d10h ago) 5d15h
laion1b-test-2-milvus-indexnode-dd48b49f6-7fd75 1/1 Running 0 4d22h
laion1b-test-2-milvus-indexnode-dd48b49f6-hc8lg 1/1 Running 0 4d22h
laion1b-test-2-milvus-indexnode-dd48b49f6-hqfwt 1/1 Running 0 4d22h
laion1b-test-2-milvus-indexnode-dd48b49f6-mgm4t 1/1 Running 0 4d22h
laion1b-test-2-milvus-indexnode-dd48b49f6-vj9g5 1/1 Running 0 4d22h
laion1b-test-2-milvus-mixcoord-576578dcbd-ctwrl 0/1 Terminating 255 (3m11s ago) 5d15h
laion1b-test-2-milvus-proxy-6b85bbb5d6-shf9d 1/1 Running 1 (5d15h ago) 5d15h
laion1b-test-2-milvus-querynode-7b584b775c-2mcxd 1/1 Running 1 (3d14h ago) 4d22h
laion1b-test-2-milvus-querynode-7b584b775c-5pddf 1/1 Running 1 (3d14h ago) 4d22h
laion1b-test-2-milvus-querynode-7b584b775c-84469 1/1 Running 0 38h
laion1b-test-2-milvus-querynode-7b584b775c-8fxsr 0/1 Error 0 4d22h
laion1b-test-2-milvus-querynode-7b584b775c-n5sls 1/1 Running 0 4d22h
laion1b-test-2-milvus-querynode-7b584b775c-njkbb 1/1 Running 0 4d22h
laion1b-test-2-milvus-querynode-7b584b775c-pldpl 1/1 Running 0 4d22h
laion1b-test-2-milvus-querynode-7b584b775c-q97vk 1/1 Running 2 (3d13h ago) 4d22h
laion1b-test-2-milvus-querynode-7b584b775c-r8lzx 1/1 Running 2 (3d11h ago) 4d22h
laion1b-test-2-milvus-querynode-7b584b775c-xcz9w 1/1 Running 0 4d22h
laion1b-test-2-milvus-querynode-7b584b775c-zpcf5 1/1 Running 0 4d22h
Anything else?
No response
Those segments is on updateSegmentIndex on every crash node.
- Node may crashed with SEGSEGV.
- Node may report some Assert error like issue metioned.
- After segment compacted, No crash on any querynode.
446930148295797152 446930148295793381 446930148295796451
2024-01-11 20:12:24 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:12:24.874 +00:00] [INFO] [datacoord/meta.go:1072] ["meta update: prepare for complete compaction mutation - complete"] [collectionID=446930148274164490] [partitionID=446930148274164493] ["new segment ID"=446930148295793381] ["new segment num of rows"=3311] ["compacted from"="[446930148295784053,446930148296136522,446930148296136891,446930148296136901,446930148296136094,446930148296137299,446930148296137183,446930148296136883,446930148296136214,446930148296136675,446930148296136943,446930148296136614,446930148296136381,446930148296137075,446930148296134235,446930148296134165,446930148295783064]"]
-- | -- | --
| | 2024-01-11 20:12:24 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:12:24.874 +00:00] [DEBUG] [datacoord/meta.go:1150] ["meta update: alter meta store for compaction updates"] ["compact from segments (segments to be updated as dropped)"="[446930148295784053,446930148296136522,446930148296136891,446930148296136901,446930148296136094,446930148296137299,446930148296137183,446930148296136883,446930148296136214,446930148296136675,446930148296136943,446930148296136614,446930148296136381,446930148296137075,446930148296134235,446930148296134165,446930148295783064]"] ["new segmentID"=446930148295793381] [binlog=14] ["stats log"=1] ["delta logs"=0] ["compact to segment"=446930148295793381]
| | 2024-01-11 20:12:25 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:12:25.264 +00:00] [INFO] [datacoord/meta.go:1176] ["meta update: alter in memory meta after compaction - complete"] ["compact to segment ID"=446930148295793381] ["compact from segment IDs"="[446930148295784053,446930148296136522,446930148296136891,446930148296136901,446930148296136094,446930148296137299,446930148296137183,446930148296136883,446930148296136214,446930148296136675,446930148296136943,446930148296136614,446930148296136381,446930148296137075,446930148296134235,446930148296134165,446930148295783064]"]
| | 2024-01-11 20:12:59 | querynodequerynode4am-node25laion1b-test-2-milvus-querynode-64f5769d6c-l8njgstdout | [2024/01/11 12:12:59.269 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164493] [segmentID=446930148295793381] [fieldID=101]
| | 2024-01-11 20:13:24 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:13:24.016 +00:00] [INFO] [datacoord/meta.go:1072] ["meta update: prepare for complete compaction mutation - complete"] [collectionID=446930148274164490] [partitionID=446930148274164496] ["new segment ID"=446930148295796451] ["new segment num of rows"=1627] ["compacted from"="[446930148296115684,446930148296111500,446930148296107327,446930148296130505,446930148296117386,446930148296136219,446930148296135402,446930148296127543,446930148296117409,446930148296114005,446930148296106347,446930148296129740,446930148296123517,446930148296122234,446930148296128936,446930148296119463,446930148296122617,446930148296117055,446930148296111221,446930148296107111,446930148296135122,446930148296130203,446930148296116527,446930148296113928,446930148296112296,446930148296109453,446930148296132806,446930148296128528,446930148296125070,446930148296124670]"]
| | 2024-01-11 20:13:24 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:13:24.016 +00:00] [DEBUG] [datacoord/meta.go:1150] ["meta update: alter meta store for compaction updates"] ["compact from segments (segments to be updated as dropped)"="[446930148296115684,446930148296111500,446930148296107327,446930148296130505,446930148296117386,446930148296136219,446930148296135402,446930148296127543,446930148296117409,446930148296114005,446930148296106347,446930148296129740,446930148296123517,446930148296122234,446930148296128936,446930148296119463,446930148296122617,446930148296117055,446930148296111221,446930148296107111,446930148296135122,446930148296130203,446930148296116527,446930148296113928,446930148296112296,446930148296109453,446930148296132806,446930148296128528,446930148296125070,446930148296124670]"] ["new segmentID"=446930148295796451] [binlog=14] ["stats log"=1] ["delta logs"=0] ["compact to segment"=446930148295796451]
| | 2024-01-11 20:13:24 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:13:24.078 +00:00] [INFO] [datacoord/meta.go:1176] ["meta update: alter in memory meta after compaction - complete"] ["compact to segment ID"=446930148295796451] ["compact from segment IDs"="[446930148296115684,446930148296111500,446930148296107327,446930148296130505,446930148296117386,446930148296136219,446930148296135402,446930148296127543,446930148296117409,446930148296114005,446930148296106347,446930148296129740,446930148296123517,446930148296122234,446930148296128936,446930148296119463,446930148296122617,446930148296117055,446930148296111221,446930148296107111,446930148296135122,446930148296130203,446930148296116527,446930148296113928,446930148296112296,446930148296109453,446930148296132806,446930148296128528,446930148296125070,446930148296124670]"]
| | 2024-01-11 20:13:33 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:13:33.998 +00:00] [INFO] [datacoord/meta.go:1072] ["meta update: prepare for complete compaction mutation - complete"] [collectionID=446930148274164490] [partitionID=446930148274164496] ["new segment ID"=446930148295797152] ["new segment num of rows"=1794] ["compacted from"="[446930148296116069,446930148296122333,446930148296120576,446930148296116483,446930148296122318,446930148296120350,446930148296110413,446930148295903383,446930148296137651,446930148296133704,446930148296120587,446930148296116032,446930148296111828,446930148296110884,446930148296136881,446930148296123704,446930148296123224,446930148296109418,446930148296130594,446930148296117092,446930148296113276,446930148296112181,446930148296129707,446930148296136092,446930148296123536,446930148296112167,446930148296111920,446930148296137302,446930148296128873,446930148296113577]"]
| | 2024-01-11 20:13:33 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:13:33.998 +00:00] [DEBUG] [datacoord/meta.go:1150] ["meta update: alter meta store for compaction updates"] ["compact from segments (segments to be updated as dropped)"="[446930148296116069,446930148296122333,446930148296120576,446930148296116483,446930148296122318,446930148296120350,446930148296110413,446930148295903383,446930148296137651,446930148296133704,446930148296120587,446930148296116032,446930148296111828,446930148296110884,446930148296136881,446930148296123704,446930148296123224,446930148296109418,446930148296130594,446930148296117092,446930148296113276,446930148296112181,446930148296129707,446930148296136092,446930148296123536,446930148296112167,446930148296111920,446930148296137302,446930148296128873,446930148296113577]"] ["new segmentID"=446930148295797152] [binlog=14] ["stats log"=1] ["delta logs"=0] ["compact to segment"=446930148295797152]
| | 2024-01-11 20:13:34 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:13:34.059 +00:00] [INFO] [datacoord/meta.go:1176] ["meta update: alter in memory meta after compaction - complete"] ["compact to segment ID"=446930148295797152] ["compact from segment IDs"="[446930148296116069,446930148296122333,446930148296120576,446930148296116483,446930148296122318,446930148296120350,446930148296110413,446930148295903383,446930148296137651,446930148296133704,446930148296120587,446930148296116032,446930148296111828,446930148296110884,446930148296136881,446930148296123704,446930148296123224,446930148296109418,446930148296130594,446930148296117092,446930148296113276,446930148296112181,446930148296129707,446930148296136092,446930148296123536,446930148296112167,446930148296111920,446930148296137302,446930148296128873,446930148296113577]"]
| | 2024-01-11 20:13:36 | querynodequerynode4am-node29laion1b-test-2-milvus-querynode-64f5769d6c-tbqpjstdout | [2024/01/11 12:13:36.589 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295796451] [fieldID=101]
| | 2024-01-11 20:13:54 | querynodequerynode4am-node29laion1b-test-2-milvus-querynode-64f5769d6c-tbqpjstdout | [2024/01/11 12:13:54.891 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164493] [segmentID=446930148295793381] [fieldID=101]
| | 2024-01-11 20:13:56 | querynodequerynode4am-node29laion1b-test-2-milvus-querynode-64f5769d6c-tbqpjstdout | [2024/01/11 12:13:56.148 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295797152] [fieldID=101]
| | 2024-01-11 20:14:27 | querynodequerynode4am-node29laion1b-test-2-milvus-querynode-64f5769d6c-tbqpjstderr | GitCommit: b46e085
| | 2024-01-11 20:17:08 | querynodequerynode4am-node29laion1b-test-2-milvus-querynode-64f5769d6c-tbqpjstdout | [2024/01/11 12:17:08.660 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295796451] [fieldID=101]
| | 2024-01-11 20:17:09 | querynodequerynode4am-node29laion1b-test-2-milvus-querynode-64f5769d6c-tbqpjstdout | [2024/01/11 12:17:09.015 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295797152] [fieldID=101]
| | 2024-01-11 20:17:45 | querynodequerynode4am-node29laion1b-test-2-milvus-querynode-64f5769d6c-tbqpjstdout | [2024/01/11 12:17:45.410 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164493] [segmentID=446930148295793381] [fieldID=101]
| | 2024-01-11 20:20:27 | querynodequerynode4am-node38laion1b-test-2-milvus-querynode-64f5769d6c-5dx6gstdout | [2024/01/11 12:20:27.220 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295797152] [fieldID=101]
| | 2024-01-11 20:20:27 | querynodequerynode4am-node38laion1b-test-2-milvus-querynode-64f5769d6c-5dx6gstdout | [2024/01/11 12:20:27.220 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164493] [segmentID=446930148295793381] [fieldID=101]
| | 2024-01-11 20:20:29 | querynodequerynode4am-node36laion1b-test-2-milvus-querynode-64f5769d6c-xnxnxstdout | [2024/01/11 12:20:29.792 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295796451] [fieldID=101]
| | 2024-01-11 20:21:03 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:21:03.310 +00:00] [INFO] [datacoord/compaction_trigger.go:662] ["generate a plan for small candidates"] ["plan segmentIDs"="[446930148295797152,446930148296343248,446930148296344098,446930148296341914,446930148296343279,446930148296341951,446930148296341965,446930148296137630,446930148296341827,446930148296128998,446930148296107655,446930148296344333,446930148296344113,446930148296133128,446930148296130152,446930148296119506,446930148296111324,446930148296108252,446930148296124903,446930148296341927,446930148296132898,446930148296122297,446930148296118089,446930148296108088,446930148296341662,446930148296135777,446930148296123725,446930148295903345,446930148296113018,446930148295903918]"] ["target segment row"=3862] ["target segment size"=12822234]
| | 2024-01-11 20:21:03 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:21:03.310 +00:00] [INFO] [datacoord/compaction_trigger.go:662] ["generate a plan for small candidates"] ["plan segmentIDs"="[446930148295796451,446930148296341984,446930148295904976,446930148296136681,446930148296122786,446930148296120730,446930148296118168,446930148296112979,446930148296128125,446930148296344212,446930148296341124,446930148296340148,446930148296120814,446930148296109551,446930148296136945,446930148296134429,446930148296112994,446930148296137518,446930148296115252,446930148296110926,446930148296123996,446930148296120537,446930148296111941,446930148296341227,446930148296134163,446930148296126865,446930148296137719,446930148296117886,446930148296344595,446930148296135159]"] ["target segment row"=4082] ["target segment size"=13659349]
| | 2024-01-11 20:21:38 | querynodequerynode4am-node38laion1b-test-2-milvus-querynode-64f5769d6c-5dx6gstderr | GitCommit: b46e085
| | 2024-01-11 20:22:16 | querynodequerynode4am-node34laion1b-test-2-milvus-querynode-64f5769d6c-b7hhkstdout | [2024/01/11 12:22:16.524 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164493] [segmentID=446930148295793381] [fieldID=101]
| | 2024-01-11 20:22:18 | querynodequerynode4am-node36laion1b-test-2-milvus-querynode-64f5769d6c-xnxnxstdout | [2024/01/11 12:22:18.643 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295797152] [fieldID=101]
| | 2024-01-11 20:23:10 | querynodequerynode4am-node16laion1b-test-2-milvus-querynode-64f5769d6c-pwbbgstdout | [2024/01/11 12:23:10.225 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295797152] [fieldID=101]
| | 2024-01-11 20:24:20 | querynodequerynode4am-node16laion1b-test-2-milvus-querynode-64f5769d6c-pwbbgstdout | [2024/01/11 12:24:20.670 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164493] [segmentID=446930148295793381] [fieldID=101]
| | 2024-01-11 20:24:20 | querynodequerynode4am-node16laion1b-test-2-milvus-querynode-64f5769d6c-pwbbgstdout | [2024/01/11 12:24:20.670 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295796451] [fieldID=101]
| | 2024-01-11 20:25:56 | querynodequerynode4am-node12laion1b-test-2-milvus-querynode-64f5769d6c-5c5jbstdout | [2024/01/11 12:25:56.119 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164493] [segmentID=446930148295793381] [fieldID=101]
| | 2024-01-11 20:26:07 | querynodequerynode4am-node34laion1b-test-2-milvus-querynode-64f5769d6c-b7hhkstdout | [2024/01/11 12:26:07.737 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295797152] [fieldID=101]
| | 2024-01-11 20:26:07 | querynodequerynode4am-node34laion1b-test-2-milvus-querynode-64f5769d6c-b7hhkstdout | [2024/01/11 12:26:07.737 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295796451] [fieldID=101]
| | 2024-01-11 20:26:42 | querynodequerynode4am-node16laion1b-test-2-milvus-querynode-64f5769d6c-pwbbgstderr | GitCommit: b46e085
| | 2024-01-11 20:28:34 | querynodequerynode4am-node16laion1b-test-2-milvus-querynode-64f5769d6c-pwbbgstdout | [2024/01/11 12:28:34.048 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295797152] [fieldID=101]
| | 2024-01-11 20:28:34 | querynodequerynode4am-node16laion1b-test-2-milvus-querynode-64f5769d6c-pwbbgstdout | [2024/01/11 12:28:34.048 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295796451] [fieldID=101]
| | 2024-01-11 20:30:30 | querynodequerynode4am-node25laion1b-test-2-milvus-querynode-64f5769d6c-l8njgstdout | [2024/01/11 12:30:30.202 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295796451] [fieldID=101]
| | 2024-01-11 20:30:30 | querynodequerynode4am-node25laion1b-test-2-milvus-querynode-64f5769d6c-l8njgstdout | [2024/01/11 12:30:30.203 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295797152] [fieldID=101]
| | 2024-01-11 20:32:47 | querynodequerynode4am-node25laion1b-test-2-milvus-querynode-64f5769d6c-l8njgstderr | GitCommit: b46e085
| | 2024-01-11 20:33:07 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:33:07.450 +00:00] [INFO] [datacoord/compaction_trigger.go:535] ["time cost of generating compaction"] ["plan ID"=446930148296562579] ["time cost"=34] [collectionID=446930148274164490] [channel=laion1b-test-2-rootcoord-dml_11_446930148274164490v1] [partitionID=446930148274164496] ["segment IDs"="[446930148295821076,446930148295684129,446930148295824216,446930148295797152,446930148295796451,446930148295795356,446930148296359948,446930148296359175,446930148296357929,446930148296349592,446930148296358288,446930148296343391,446930148296359462,446930148296359024,446930148296125538,446930148296344488,446930148296122478,446930148296340940,446930148296354119,446930148296125472,446930148296361242,446930148296120964,446930148296344057,446930148296349242,446930148296358974,446930148296123073,446930148296359209,446930148296349437,446930148296340505,446930148296127615]"]
| | 2024-01-11 20:34:04 | querynodequerynode4am-node17laion1b-test-2-milvus-querynode-64f5769d6c-7kznrstdout | [2024/01/11 12:34:04.127 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295796451] [fieldID=101]
| | 2024-01-11 20:34:04 | querynodequerynode4am-node17laion1b-test-2-milvus-querynode-64f5769d6c-7kznrstdout | [2024/01/11 12:34:04.864 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295797152] [fieldID=101]
| | 2024-01-11 20:35:27 | querynodequerynode4am-node16laion1b-test-2-milvus-querynode-64f5769d6c-pwbbgstderr | GitCommit: b46e085
| | 2024-01-11 20:36:16 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:36:16.539 +00:00] [INFO] [datacoord/meta.go:1072] ["meta update: prepare for complete compaction mutation - complete"] [collectionID=446930148274164490] [partitionID=446930148274164496] ["new segment ID"=446930148295852084] ["new segment num of rows"=32929] ["compacted from"="[446930148295821076,446930148295684129,446930148295824216,446930148295797152,446930148295796451,446930148295795356,446930148296359948,446930148296359175,446930148296357929,446930148296349592,446930148296358288,446930148296343391,446930148296359462,446930148296359024,446930148296125538,446930148296344488,446930148296122478,446930148296340940,446930148296354119,446930148296125472,446930148296361242,446930148296120964,446930148296344057,446930148296349242,446930148296358974,446930148296123073,446930148296359209,446930148296349437,446930148296340505,446930148296127615]"]
| | 2024-01-11 20:36:16 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:36:16.541 +00:00] [DEBUG] [datacoord/meta.go:1150] ["meta update: alter meta store for compaction updates"] ["compact from segments (segments to be updated as dropped)"="[446930148295821076,446930148295684129,446930148295824216,446930148295797152,446930148295796451,446930148295795356,446930148296359948,446930148296359175,446930148296357929,446930148296349592,446930148296358288,446930148296343391,446930148296359462,446930148296359024,446930148296125538,446930148296344488,446930148296122478,446930148296340940,446930148296354119,446930148296125472,446930148296361242,446930148296120964,446930148296344057,446930148296349242,446930148296358974,446930148296123073,446930148296359209,446930148296349437,446930148296340505,446930148296127615]"] ["new segmentID"=446930148295852084] [binlog=14] ["stats log"=1] ["delta logs"=2] ["compact to segment"=446930148295852084]
| | 2024-01-11 20:36:17 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:36:17.272 +00:00] [INFO] [datacoord/meta.go:1176] ["meta update: alter in memory meta after compaction - complete"] ["compact to segment ID"=446930148295852084] ["compact from segment IDs"="[446930148295821076,446930148295684129,446930148295824216,446930148295797152,446930148295796451,446930148295795356,446930148296359948,446930148296359175,446930148296357929,446930148296349592,446930148296358288,446930148296343391,446930148296359462,446930148296359024,446930148296125538,446930148296344488,446930148296122478,446930148296340940,446930148296354119,446930148296125472,446930148296361242,446930148296120964,446930148296344057,446930148296349242,446930148296358974,446930148296123073,446930148296359209,446930148296349437,446930148296340505,446930148296127615]"]
| | 2024-01-11 20:37:53 | querynodequerynode4am-node17laion1b-test-2-milvus-querynode-64f5769d6c-7kznrstderr | GitCommit: b46e085
| | 2024-01-11 20:38:20 | querynodequerynode4am-node16laion1b-test-2-milvus-querynode-64f5769d6c-pwbbgstdout | [2024/01/11 12:38:20.453 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295796451] [fieldID=101]
| | 2024-01-11 20:38:20 | querynodequerynode4am-node16laion1b-test-2-milvus-querynode-64f5769d6c-pwbbgstdout | [2024/01/11 12:38:20.502 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164496] [segmentID=446930148295797152] [fieldID=101]
| | 2024-01-11 20:39:42 | querynodequerynode4am-node25laion1b-test-2-milvus-querynode-64f5769d6c-l8njgstdout | [2024/01/11 12:39:42.384 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164493] [segmentID=446930148295793381] [fieldID=101]
| | 2024-01-11 20:42:13 | querynodequerynode4am-node16laion1b-test-2-milvus-querynode-64f5769d6c-pwbbgstderr | GitCommit: b46e085
| | 2024-01-11 20:57:24 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 12:57:24.369 +00:00] [INFO] [datacoord/compaction_trigger.go:910] ["total delete entities is too much, trigger compaction"] [segmentID=446930148295793381] [numRows=3311] ["deleted rows"=1278] ["delete log size"=24180]
| | 2024-01-11 21:02:12 | querynodequerynode4am-node34laion1b-test-2-milvus-querynode-64f5769d6c-b7hhkstderr | GitCommit: b46e085
| | 2024-01-11 21:05:24 | querynodequerynode4am-node35laion1b-test-2-milvus-querynode-64f5769d6c-bm4vpstdout | [2024/01/11 13:05:24.065 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164493] [segmentID=446930148295793381] [fieldID=101]
| | 2024-01-11 21:09:28 | querynodequerynode4am-node25laion1b-test-2-milvus-querynode-64f5769d6c-l8njgstdout | [2024/01/11 13:09:28.434 +00:00] [INFO] [segments/segment.go:938] ["updateSegmentIndex done"] [collectionID=446930148274164490] [partitionID=446930148274164493] [segmentID=446930148295793381] [fieldID=101]
| | 2024-01-11 21:09:42 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 13:09:42.663 +00:00] [INFO] [datacoord/meta.go:1072] ["meta update: prepare for complete compaction mutation - complete"] [collectionID=446930148274164490] [partitionID=446930148274164493] ["new segment ID"=446930148296819389] ["new segment num of rows"=330805] ["compacted from"="[446930148295778548,446930148295790712,446930148295793381,446930148296562651,446930148296359953,446930148296987764,446930148296359173,446930148296777865,446930148296574262,446930148296137792,446930148296361764,446930148296573188,446930148296357912,446930148296779181,446930148296986754,446930148296565464,446930148296983470,446930148296344471,446930148296568791,446930148296359459,446930148296570223,446930148296568772,446930148296571602,446930148296575728,446930148296986773,446930148296349594,446930148296570036,446930148296571881,446930148296343395,446930148296776393]"]
| | 2024-01-11 21:09:42 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 13:09:42.664 +00:00] [DEBUG] [datacoord/meta.go:1150] ["meta update: alter meta store for compaction updates"] ["compact from segments (segments to be updated as dropped)"="[446930148295778548,446930148295790712,446930148295793381,446930148296562651,446930148296359953,446930148296987764,446930148296359173,446930148296777865,446930148296574262,446930148296137792,446930148296361764,446930148296573188,446930148296357912,446930148296779181,446930148296986754,446930148296565464,446930148296983470,446930148296344471,446930148296568791,446930148296359459,446930148296570223,446930148296568772,446930148296571602,446930148296575728,446930148296986773,446930148296349594,446930148296570036,446930148296571881,446930148296343395,446930148296776393]"] ["new segmentID"=446930148296819389] [binlog=14] ["stats log"=1] ["delta logs"=3] ["compact to segment"=446930148296819389]
| | 2024-01-11 21:10:07 | mixcoordmixcoord4am-node17laion1b-test-2-milvus-mixcoord-59f5cd484c-5bfz6stdout | [2024/01/11 13:10:07.286 +00:00] [INFO] [datacoord/meta.go:1176] ["meta update: alter in memory meta after compaction - complete"] ["compact to segment ID"=446930148296819389] ["compact from segment IDs"="[446930148295778548,446930148295790712,446930148295793381,446930148296562651,446930148296359953,446930148296987764,446930148296359173,446930148296777865,446930148296574262,446930148296137792,446930148296361764,446930148296573188,446930148296357912,446930148296779181,446930148296986754,446930148296565464,446930148296983470,446930148296344471,446930148296568791,446930148296359459,446930148296570223,446930148296568772,446930148296571602,446930148296575728,446930148296986773,446930148296349594,446930148296570036,446930148296571881,446930148296343395,446930148296776393]"]
