milvus-cdc icon indicating copy to clipboard operation
milvus-cdc copied to clipboard

Published 20 hours ago • verified publisher icon zilliztech

When the CDC stops the synchronization task, the data cannot be queried in the target environment.

Open huangping11 opened this issue 8 months ago • 16 comments

Current Behavior

[root@wy3-db241 milvus-cdc]# curl -X POST http://localhost:8444/cdc \

-H "Content-Type: application/json"
-d '{ "request_type": "create", "request_data": { "milvus_connect_param": { "uri": "http://192.168.3.242:19530", "token":"root:Milvus", "connect_timeout": 10 }, "db_collections": { "": [ { "name": "" } ] } } }' {"code":200,"data":{"task_id":"847723f393fa4602893831c5001a006e"}} [root@wy3-db241 milvus-cdc]#

After the CDC (Change Data Capture) synchronization configuration is completed, newly inserted data has been successfully synchronized to the target environment, and queries can be performed in the target environment. However, once the synchronization task is stopped, the data cannot be queried.

source:

Image

target:

Image

It can be seen that the data in the target environment is consistent with that in the source environment.

Expected Behavior

No response

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

huangping11 avatar Mar 13 '25 08:03 huangping11

stops the synchronization task [root@wy3-db241 milvus-cdc]# curl -X POST -H "Content-Type: application/json" -d '{

"request_type":"pause", "request_data": { "task_id": "847723f393fa4602893831c5001a006e" } }' http://localhost:8444/cdc {"code":200} [root@wy3-db241 milvus-cdc]#

targer milvus log [2025/03/13 08:55:31.152 +00:00] [INFO] [observers/target_observer.go:540] ["observer trigger update current target"] [collectionID=456616994539308425] [2025/03/13 08:55:31.153 +00:00] [INFO] [datacoord/services.go:818] ["get recovery info request received"] [traceID=3b9fd303b35e9462751a74efc1bc1071] [collectionID=456616994539308425] [partitionIDs="[]"] [2025/03/13 08:55:31.153 +00:00] [INFO] [datacoord/handler.go:237] [GetQueryVChanPositions] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"="{"456616994539308426":0}"] [2025/03/13 08:55:31.153 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [posTs=456617131700649985] [posTime=2025/03/13 08:55:23.736 +00:00] [2025/03/13 08:55:31.153 +00:00] [INFO] [datacoord/services.go:833] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=3b9fd303b35e9462751a74efc1bc1071] [collectionID=456616994539308425] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] [2025/03/13 08:55:40.550 +00:00] [INFO] [datacoord/compaction_trigger.go:323] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=456616994540509834] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0] [2025/03/13 08:55:40.551 +00:00] [INFO] [datacoord/compaction_policy_l0.go:33] ["start trigger l0CompactionPolicy..."] [2025/03/13 08:55:40.551 +00:00] [INFO] [datacoord/compaction_policy_single.go:49] ["start trigger singleCompactionPolicy..."] [2025/03/13 08:55:40.551 +00:00] [INFO] [datacoord/compaction_policy_single.go:69] ["start trigger single compaction"] [collectionID=456616994539308425] [2025/03/13 08:55:40.552 +00:00] [INFO] [datacoord/compaction_policy_single.go:125] ["finish trigger single compaction"] [collectionID=456616994539308425] [viewNum=0] [2025/03/13 08:55:41.151 +00:00] [INFO] [querynodev2/services.go:1338] ["sync action"] [traceID=32c5e36c3def56883c59e2ccf520596a] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1741856131154224458] [2025/03/13 08:55:41.151 +00:00] [INFO] [delegator/distribution.go:297] ["Update readable segment version"] [oldVersion=1741856121153628978] [newVersion=1741856131154224458] [growingSegmentNum=1] [sealedSegmentNum=0] [2025/03/13 08:55:41.152 +00:00] [INFO] [observers/target_observer.go:540] ["observer trigger update current target"] [collectionID=456616994539308425] [2025/03/13 08:55:41.153 +00:00] [INFO] [datacoord/services.go:818] ["get recovery info request received"] [traceID=d761acf79004c3e4d1d5de5b9fa458f1] [collectionID=456616994539308425] [partitionIDs="[]"] [2025/03/13 08:55:41.153 +00:00] [INFO] [datacoord/handler.go:237] [GetQueryVChanPositions] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"="{"456616994539308426":0}"] [2025/03/13 08:55:41.153 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [posTs=456617131700649985] [posTime=2025/03/13 08:55:23.736 +00:00] [2025/03/13 08:55:41.153 +00:00] [INFO] [datacoord/services.go:833] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=d761acf79004c3e4d1d5de5b9fa458f1] [collectionID=456616994539308425] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] [2025/03/13 08:55:51.152 +00:00] [INFO] [querynodev2/services.go:1338] ["sync action"] [traceID=8e8bf53e051f52d40f72c09dd32b45fd] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1741856141154072383] [2025/03/13 08:55:51.152 +00:00] [INFO] [delegator/distribution.go:297] ["Update readable segment version"] [oldVersion=1741856131154224458] [newVersion=1741856141154072383] [growingSegmentNum=1] [sealedSegmentNum=0] [2025/03/13 08:55:51.152 +00:00] [INFO] [observers/target_observer.go:540] ["observer trigger update current target"] [collectionID=456616994539308425] [2025/03/13 08:55:51.153 +00:00] [INFO] [datacoord/services.go:818] ["get recovery info request received"] [traceID=adaf5e2e30d625fb181868cb3af1e24d] [collectionID=456616994539308425] [partitionIDs="[]"] [2025/03/13 08:55:51.153 +00:00] [INFO] [datacoord/handler.go:237] [GetQueryVChanPositions] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"="{"456616994539308426":0}"] [2025/03/13 08:55:51.153 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [posTs=456617131700649985] [posTime=2025/03/13 08:55:23.736 +00:00] [2025/03/13 08:55:51.153 +00:00] [INFO] [datacoord/services.go:833] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=adaf5e2e30d625fb181868cb3af1e24d] [collectionID=456616994539308425] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] [2025/03/13 08:56:01.152 +00:00] [INFO] [querynodev2/services.go:1338] ["sync action"] [traceID=23cb4121d6371939d09d9b0c2af14382] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1741856151154101831] [2025/03/13 08:56:01.152 +00:00] [INFO] [delegator/distribution.go:297] ["Update readable segment version"] [oldVersion=1741856141154072383] [newVersion=1741856151154101831] [growingSegmentNum=1] [sealedSegmentNum=0] [2025/03/13 08:56:01.153 +00:00] [INFO] [observers/target_observer.go:540] ["observer trigger update current target"] [collectionID=456616994539308425] [2025/03/13 08:56:01.154 +00:00] [INFO] [datacoord/services.go:818] ["get recovery info request received"] [traceID=7a19ec00627c5fcc2a1e52897ec3f50c] [collectionID=456616994539308425] [partitionIDs="[]"] [2025/03/13 08:56:01.154 +00:00] [INFO] [datacoord/handler.go:237] [GetQueryVChanPositions] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"="{"456616994539308426":0}"] [2025/03/13 08:56:01.154 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [posTs=456617131700649985] [posTime=2025/03/13 08:55:23.736 +00:00] [2025/03/13 08:56:01.154 +00:00] [INFO] [datacoord/services.go:833] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=7a19ec00627c5fcc2a1e52897ec3f50c] [collectionID=456616994539308425] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] [2025/03/13 08:56:10.390 +00:00] [INFO] [msgstream/mq_msgstream.go:218] ["start to close mq msg stream"] ["producer num"=1] ["consumer num"=0] [2025/03/13 08:56:11.151 +00:00] [INFO] [querynodev2/services.go:1338] ["sync action"] [traceID=d2333bc7bf895706a445d792afe20a8a] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1741856161155208495] [2025/03/13 08:56:11.151 +00:00] [INFO] [delegator/distribution.go:297] ["Update readable segment version"] [oldVersion=1741856151154101831] [newVersion=1741856161155208495] [growingSegmentNum=1] [sealedSegmentNum=0] [2025/03/13 08:56:11.152 +00:00] [INFO] [observers/target_observer.go:540] ["observer trigger update current target"] [collectionID=456616994539308425] [2025/03/13 08:56:11.152 +00:00] [INFO] [datacoord/services.go:818] ["get recovery info request received"] [traceID=4217aefbd0c21d75b919f2574cb53869] [collectionID=456616994539308425] [partitionIDs="[]"] [2025/03/13 08:56:11.152 +00:00] [INFO] [datacoord/handler.go:237] [GetQueryVChanPositions] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"="{"456616994539308426":0}"] [2025/03/13 08:56:11.153 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [posTs=456617131700649985] [posTime=2025/03/13 08:55:23.736 +00:00] [2025/03/13 08:56:11.153 +00:00] [INFO] [datacoord/services.go:833] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=4217aefbd0c21d75b919f2574cb53869] [collectionID=456616994539308425] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] [2025/03/13 08:56:21.151 +00:00] [INFO] [querynodev2/services.go:1338] ["sync action"] [traceID=36e32fadfdb2f67afcb2b1d27e96bbc8] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1741856171153488636] [2025/03/13 08:56:21.151 +00:00] [INFO] [delegator/distribution.go:297] ["Update readable segment version"] [oldVersion=1741856161155208495] [newVersion=1741856171153488636] [growingSegmentNum=1] [sealedSegmentNum=0] [2025/03/13 08:56:21.152 +00:00] [INFO] [observers/target_observer.go:540] ["observer trigger update current target"] [collectionID=456616994539308425] [2025/03/13 08:56:21.153 +00:00] [INFO] [datacoord/services.go:818] ["get recovery info request received"] [traceID=3fa12cf01ec49202172d0dd6f6dd3380] [collectionID=456616994539308425] [partitionIDs="[]"] [2025/03/13 08:56:21.153 +00:00] [INFO] [datacoord/handler.go:237] [GetQueryVChanPositions] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"="{"456616994539308426":0}"] [2025/03/13 08:56:21.153 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [posTs=456617131700649985] [posTime=2025/03/13 08:55:23.736 +00:00] [2025/03/13 08:56:21.153 +00:00] [INFO] [datacoord/services.go:833] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=3fa12cf01ec49202172d0dd6f6dd3380] [collectionID=456616994539308425] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] [2025/03/13 08:56:31.152 +00:00] [INFO] [querynodev2/services.go:1338] ["sync action"] [traceID=38aedd62fe0e6f5e47627e6441e7b953] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1741856181153650205] [2025/03/13 08:56:31.152 +00:00] [INFO] [delegator/distribution.go:297] ["Update readable segment version"] [oldVersion=1741856171153488636] [newVersion=1741856181153650205] [growingSegmentNum=1] [sealedSegmentNum=0] [2025/03/13 08:56:31.152 +00:00] [INFO] [observers/target_observer.go:540] ["observer trigger update current target"] [collectionID=456616994539308425] [2025/03/13 08:56:31.153 +00:00] [INFO] [datacoord/services.go:818] ["get recovery info request received"] [traceID=5910123b959ffdb710323d21151983aa] [collectionID=456616994539308425] [partitionIDs="[]"] [2025/03/13 08:56:31.154 +00:00] [INFO] [datacoord/handler.go:237] [GetQueryVChanPositions] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"="{"456616994539308426":0}"] [2025/03/13 08:56:31.154 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [posTs=456617131700649985] [posTime=2025/03/13 08:55:23.736 +00:00] [2025/03/13 08:56:31.154 +00:00] [INFO] [datacoord/services.go:833] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=5910123b959ffdb710323d21151983aa] [collectionID=456616994539308425] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0] [2025/03/13 08:56:40.551 +00:00] [WARN] [datacoord/session_manager.go:251] ["failed to sync segments after retry"] [nodeID=1] [planID=0] [error="context canceled"] [2025/03/13 08:56:40.551 +00:00] [WARN] [datacoord/sync_segments_scheduler.go:157] ["fail to sync segments with node"] [collectionID=456616994539308425] [partitionID=456616994539308426] [channelName=by-dev-rootcoord-dml_0_456616994539308425v0] [nodeID=1] [error="context canceled"] [2025/03/13 08:56:40.551 +00:00] [WARN] [datacoord/sync_segments_scheduler.go:109] ["sync segment with channel failed, retry next ticker"] [collectionID=456616994539308425] [partitionID=456616994539308426] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [error="context canceled"] [2025/03/13 08:56:40.551 +00:00] [INFO] [datacoord/import_checker.go:130] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0] [2025/03/13 08:56:40.551 +00:00] [INFO] [datacoord/import_checker.go:130] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0] [2025/03/13 08:56:40.551 +00:00] [INFO] [datacoord/compaction_policy_l0.go:33] ["start trigger l0CompactionPolicy..."] [2025/03/13 08:56:40.551 +00:00] [INFO] [datacoord/compaction_policy_single.go:49] ["start trigger singleCompactionPolicy..."] [2025/03/13 08:56:40.551 +00:00] [INFO] [datacoord/compaction_policy_single.go:69] ["start trigger single compaction"] [collectionID=456616994539308425] [2025/03/13 08:56:40.553 +00:00] [INFO] [datacoord/compaction_policy_single.go:125] ["finish trigger single compaction"] [collectionID=456616994539308425] [viewNum=0] [2025/03/13 08:56:40.553 +00:00] [INFO] [datacoord/compaction_trigger.go:323] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=456616994540510143] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0] [2025/03/13 08:56:40.648 +00:00] [WARN] [timerecord/time_recorder.go:134] ["RootCoord haven't synchronized the time tick for 2.000000 minutes"] [2025/03/13 08:56:41.133 +00:00] [INFO] [importv2/util.go:224] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0] [2025/03/13 08:56:41.134 +00:00] [INFO] [importv2/util.go:224] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0] [2025/03/13 08:56:41.151 +00:00] [INFO] [querynodev2/services.go:1338] ["sync action"] [traceID=2c3b66f941662ad8cf9da08983b514f6] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1741856191154744157] [2025/03/13 08:56:41.151 +00:00] [INFO] [delegator/distribution.go:297] ["Update readable segment version"] [oldVersion=1741856181153650205] [newVersion=1741856191154744157] [growingSegmentNum=1] [sealedSegmentNum=0] [2025/03/13 08:56:41.151 +00:00] [INFO] [observers/target_observer.go:540] ["observer trigger update current target"] [collectionID=456616994539308425] [2025/03/13 08:56:41.152 +00:00] [INFO] [datacoord/services.go:818] ["get recovery info request received"] [traceID=29286505b87fcd07f74a76d1b3ece93a] [collectionID=456616994539308425] [partitionIDs="[]"] [2025/03/13 08:56:41.152 +00:00] [INFO] [datacoord/handler.go:237] [GetQueryVChanPositions] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"="{"456616994539308426":0}"] [2025/03/13 08:56:41.152 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [posTs=456617131700649985] [posTime=2025/03/13 08:55:23.736 +00:00] [2025/03/13 08:56:41.152 +00:00] [INFO] [datacoord/services.go:833] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=29286505b87fcd07f74a76d1b3ece93a] [collectionID=456

huangping11 avatar Mar 13 '25 08:03 huangping11

Refresh the ATTU page. No data can be queried.

Image

The following is the log output of the target environment.

[2025/03/13 08:58:27.074 +00:00] [WARN] [delegator/delegator.go:482] ["delegator query failed to wait tsafe"] [traceID=dc423c5cd690d31f6c14f055a75e02d4] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [replicaID=456616994701049857] [error="context deadline exceeded"] [2025/03/13 08:58:27.074 +00:00] [WARN] [querynodev2/handlers.go:223] ["failed to query on delegator"] [traceID=dc423c5cd690d31f6c14f055a75e02d4] [msgID=456617175822893063] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [scope=All] [error="context deadline exceeded"] [2025/03/13 08:58:27.074 +00:00] [WARN] [proxy/task_query.go:565] ["QueryNode query result error"] [traceID=dc423c5cd690d31f6c14f055a75e02d4] [collection=456616994539308425] [partitionIDs="[]"] [nodeID=1] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [errorCode=UnexpectedError] [reason="context deadline exceeded"] [2025/03/13 08:58:27.074 +00:00] [WARN] [proxy/lb_policy.go:197] ["search/query channel failed"] [collectionID=456616994539308425] [channelName=by-dev-rootcoord-dml_0_456616994539308425v0] [nodeID=1] [error="fail to Query on QueryNode 1: context deadline exceeded"] [errorVerbose="fail to Query on QueryNode 1: context deadline exceeded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*queryTask).queryShard\n | \t/workspace/source/internal/proxy/task_query.go:566\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:195\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:164\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n | \t/workspace/source/internal/proxy/lb_policy.go:232\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) fail to Query on QueryNode 1\nWraps: (3) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [2025/03/13 08:58:27.074 +00:00] [WARN] [proxy/impl.go:3473] ["Query failed to WaitToFinish"] [traceID=dc423c5cd690d31f6c14f055a75e02d4] [role=proxy] [db=db1] [collection=t1] [partitions="[]"] [ConsistencyLevel=Strong] [useDefaultConsistency=false] [error="proxy TaskCondition context Done: context deadline exceeded"] [errorVerbose="proxy TaskCondition context Done: context deadline exceeded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*TaskCondition).WaitToFinish\n | \t/workspace/source/internal/proxy/condition.go:45\n | github.com/milvus-io/milvus/internal/proxy.(*Proxy).query\n | \t/workspace/source/internal/proxy/impl.go:3472\n | github.com/milvus-io/milvus/internal/proxy.(*Proxy).Query\n | \t/workspace/source/internal/proxy/impl.go:3561\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Query\n | \t/workspace/source/internal/distributed/proxy/service.go:858\n | github.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Query_Handler.func1\n | \t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/[email protected]/milvuspb/milvus_grpc.pb.go:2063\n | github.com/milvus-io/milvus/internal/proxy/connection.KeepActiveInterceptor\n | \t/workspace/source/internal/proxy/connection/util.go:60\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n | github.com/milvus-io/milvus/internal/proxy.TraceLogInterceptor\n | \t/workspace/source/internal/proxy/trace_log_interceptor.go:36\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n | github.com/milvus-io/milvus/internal/proxy/accesslog.UnaryUpdateAccessInfoInterceptor\n | \t/workspace/source/internal/proxy/accesslog/util.go:47\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.RateLimitInterceptor.func5\n | \t/workspace/source/internal/proxy/rate_limit_interceptor.go:56\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n | github.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n | \t/workspace/source/pkg/util/logutil/grpc_interceptor.go:23\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func4\n | \t/workspace/source/internal/proxy/privilege_interceptor.go:112\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n | github.com/milvus-io/milvus/internal/proxy.HookInterceptor\n | \t/workspace/source/internal/proxy/hook_interceptor.go:56\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerHookInterceptor.func3\n | \t/workspace/source/internal/proxy/hook_interceptor.go:21\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.DatabaseInterceptor.func2\n | \t/workspace/source/internal/proxy/database_interceptor.go:15\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:47\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n | github.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLogInterceptor\n | \t/workspace/source/internal/proxy/accesslog/util.go:38\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6\n | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34\n | github.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Query_Handler\n | \t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/[email protected]/milvuspb/milvus_grpc.pb.go:2065\n | google.golang.org/grpc.(*Server).processUnaryRPC\n | \t/go/pkg/mod/google.golang.org/[email protected]/server.go:1343\n | google.golang.org/grpc.(*Server).handleStream\n | \t/go/pkg/mod/google.golang.org/[email protected]/server.go:1737\n | google.golang.org/grpc.(*Server).serveStreams.func1.1\n | \t/go/pkg/mod/google.golang.org/[email protected]/server.go:986\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) proxy TaskCondition context Done\nWraps: (3) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) context.deadlineExceededError"] [2025/03/13 08:58:27.075 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [traceID=dc423c5cd690d31f6c14f055a75e02d4] [retried=0] [error="failed to search/query delegator 1 for channel by-dev-rootcoord-dml_0_456616994539308425v0: fail to Query on QueryNode 1: context deadline exceeded"] [errorVerbose="failed to search/query delegator 1 for channel by-dev-rootcoord-dml_0_456616994539308425v0: fail to Query on QueryNode 1: context deadline exceeded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:203\n | [...repeated from below...]\nWraps: (2) failed to search/query delegator 1 for channel by-dev-rootcoord-dml_0_456616994539308425v0\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*queryTask).queryShard\n | \t/workspace/source/internal/proxy/task_query.go:566\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:195\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:164\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n | \t/workspace/source/internal/proxy/lb_policy.go:232\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (4) fail to Query on QueryNode 1\nWraps: (5) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) errutil.withPrefix (5) merr.milvusError"] [2025/03/13 08:58:27.075 +00:00] [INFO] [proxy/impl.go:3434] ["Query slow"] [traceID=dc423c5cd690d31f6c14f055a75e02d4] [role=proxy] [db=db1] [collection=t1] [partitions="[]"] [ConsistencyLevel=Strong] [useDefaultConsistency=false] [expr=] [OutputFields="[count()]"] [travel_timestamp=0] [guarantee_timestamp=456617175822893063] [duration=15.001603252s] [2025/03/13 08:58:27.075 +00:00] [WARN] [proxy/task_query.go:470] ["fail to execute query"] [traceID=dc423c5cd690d31f6c14f055a75e02d4] [collection=456616994539308425] [partitionIDs="[]"] [requestType=query] [error="failed to search/query delegator 1 for channel by-dev-rootcoord-dml_0_456616994539308425v0: fail to Query on QueryNode 1: context deadline exceeded"] [errorVerbose="failed to search/query delegator 1 for channel by-dev-rootcoord-dml_0_456616994539308425v0: fail to Query on QueryNode 1: context deadline exceeded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:203\n | [...repeated from below...]\nWraps: (2) failed to search/query delegator 1 for channel by-dev-rootcoord-dml_0_456616994539308425v0\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*queryTask).queryShard\n | \t/workspace/source/internal/proxy/task_query.go:566\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:195\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:164\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n | \t/workspace/source/internal/proxy/lb_policy.go:232\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (4) fail to Query on QueryNode 1\nWraps: (5) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) merr.milvusError"] [2025/03/13 08:58:27.076 +00:00] [WARN] [proxy/task_scheduler.go:477] ["Failed to execute task: "] [traceID=dc423c5cd690d31f6c14f055a75e02d4] [error="failed to query: failed to search/query delegator 1 for channel by-dev-rootcoord-dml_0_456616994539308425v0: fail to Query on QueryNode 1: context deadline exceeded"] [errorVerbose="failed to query: failed to search/query delegator 1 for channel by-dev-rootcoord-dml_0_456616994539308425v0: fail to Query on QueryNode 1: context deadline exceeded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n | \t/workspace/source/internal/proxy/task_query.go:471\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/workspace/source/internal/proxy/task_scheduler.go:474\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n | \t/workspace/source/internal/proxy/task_scheduler.go:553\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\nWraps: (2) failed to query\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:203\n | [...repeated from below...]\nWraps: (4) failed to search/query delegator 1 for channel by-dev-rootcoord-dml_0_456616994539308425v0\nWraps: (5) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*queryTask).queryShard\n | \t/workspace/source/internal/proxy/task_query.go:566\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:195\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:164\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n | \t/workspace/source/internal/proxy/lb_policy.go:232\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (6) fail to Query on QueryNode 1\nWraps: (7) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) merr.milvusError"] [2025/03/13 08:58:27.079 +00:00] [INFO] [querycoordv2/services.go:882] ["get replicas request received"] [traceID=b677abddaf881478e6d62e2185c9302c] [collectionID=456616994539308425] [with-shard-nodes=false] [2025/03/13 08:58:31.151 +00:00] [INFO] [querynodev2/services.go:1338] ["sync action"] [traceID=34f6fb77606f9012bb979d44ee9f7e3b] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1741856301156063188] [2025/03/13 08:58:31.151 +00:00] [INFO] [delegator/distribution.go:297] ["Update readable segment version"] [oldVersion=1741856291154633369] [newVersion=1741856301156063188] [growingSegmentNum=1] [sealedSegmentNum=0] [2025/03/13 08:58:31.151 +00:00] [INFO] [observers/target_observer.go:540] ["observer trigger update current target"] [collectionID=456616994539308425] [2025/03/13 08:58:31.152 +00:00] [INFO] [datacoord/services.go:818] ["get recovery info request received"] [traceID=f51d09b385146a8c05103202b94d456c] [collectionID=456616994539308425] [partitionIDs="[]"] [2025/03/13 08:58:31.152 +00:00] [INFO] [datacoord/handler.go:237] [GetQueryVChanPositions] [collectionID=456616994539308425] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [numOfSegments=1] ["result flushed"=0] ["result growing"=1] ["result L0"=0] ["partition stats"="{"456616994539308426":0}"] [2025/03/13 08:58:31.152 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] [posTs=456617131700649985] [posTime=2025/03/13 08:55:23.736 +00:00] [2025/03/13 08:58:31.153 +00:00] [INFO] [datacoord/services.go:833] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=f51d09b385146a8c05103202b94d456c] [collectionID=456616994539308425] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_456616994539308425v0] ["# of unflushed segments"=1] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]

huangping11 avatar Mar 13 '25 09:03 huangping11

Restart the synchronization task.

No data can be queried.

[root@wy3-db241 milvus-cdc]# curl -X POST -H "Content-Type: application/json" -d '{

"request_type":"resume", "request_data": { "task_id": "847723f393fa4602893831c5001a006e" } }' http://localhost:8444/cdc {"code":200}

Image

The CDC log output is as follows.

[2025/03/13 17:00:57.147 +08:00] [INFO] [server/server.go:131] ["request receive"] [type=resume] [data="{"task_id":"847723f393fa4602893831c5001a006e"}"] [2025/03/13 17:00:57.155 +08:00] [INFO] [util/channel_mapping.go:58] [ChannelMapping] [sourceCnt=0] [targetCnt=0] [averageCnt=1] [2025/03/13 17:00:57.163 +08:00] [INFO] [reader/etcd_op.go:777] ["current time"] [ts=2025/03/13 17:00:57.986 +08:00] [2025/03/13 17:00:57.167 +08:00] [INFO] [reader/etcd_op.go:566] ["get all collection data"] [count=2] [2025/03/13 17:00:57.169 +08:00] [INFO] [reader/etcd_op.go:710] ["get all partition data"] [partition_num=2] [2025/03/13 17:00:57.169 +08:00] [INFO] [writer/channel_writer.go:87] ["new channel writer"] [droppedObjs="{"database":{},"collection":{},"partition":{}}"] [2025/03/13 17:00:57.172 +08:00] [INFO] [msgdispatcher/manager.go:67] ["create new dispatcherManager"] [role=cdc-fd31c6b2e190440fb80a9683b6a83aa3] [nodeID=8444] [pchannel=by-dev-replicate-msg] [2025/03/13 17:00:57.173 +08:00] [INFO] [msgdispatcher/dispatcher.go:96] ["creating dispatcher..."] [pchannel=by-dev-replicate-msg] [subName=cdc-fd31c6b2e190440fb80a9683b6a83aa3-8444-by-dev-replicate-msg_847723f393fa4602893831c5001a006ev0-true] [isMain=true] [2025/03/13 17:00:57.173 +08:00] [INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true] [2025/03/13 17:00:57.173 +08:00] [INFO] [msgdispatcher/manager.go:178] ["dispatcherManager is running..."] [role=cdc-fd31c6b2e190440fb80a9683b6a83aa3] [nodeID=8444] [pchannel=by-dev-replicate-msg] [2025/03/13 17:00:57.194 +08:00] [INFO] [msgstream/mq_msgstream.go:214] ["Successfully create consumer"] [channel=by-dev-replicate-msg] [subname=cdc-fd31c6b2e190440fb80a9683b6a83aa3-8444-by-dev-replicate-msg_847723f393fa4602893831c5001a006ev0-true] [2025/03/13 17:00:57.195 +08:00] [INFO] [msgstream/mq_msgstream.go:535] ["MsgStream seek begin"] [channel=by-dev-replicate-msg] [MessageID="CBIQChgAIAA="] [includeCurrentMsg=false] [2025/03/13 17:00:57.303 +08:00] [INFO] [msgstream/mq_msgstream.go:541] ["MsgStream seek finished"] [channel=by-dev-replicate-msg] [2025/03/13 17:00:57.303 +08:00] [INFO] [msgdispatcher/dispatcher.go:125] ["seek successfully"] [pchannel=by-dev-replicate-msg] [subName=cdc-fd31c6b2e190440fb80a9683b6a83aa3-8444-by-dev-replicate-msg_847723f393fa4602893831c5001a006ev0-true] [isMain=true] [posTs=0] [posTime=1970/01/01 08:00:00.000 +08:00] [tsLag=483849h0m57.303778615s] [2025/03/13 17:00:57.303 +08:00] [INFO] [msgdispatcher/dispatcher.go:161] ["add new target"] [vchannel=by-dev-replicate-msg_847723f393fa4602893831c5001a006ev0] [isMain=true] [2025/03/13 17:00:57.303 +08:00] [INFO] [msgdispatcher/manager.go:115] ["add main dispatcher"] [role=cdc-fd31c6b2e190440fb80a9683b6a83aa3] [nodeID=8444] [vchannel=by-dev-replicate-msg_847723f393fa4602893831c5001a006ev0] [2025/03/13 17:00:57.303 +08:00] [INFO] [msgdispatcher/dispatcher.go:189] ["get signal"] [pchannel=by-dev-replicate-msg] [signal=start] [isMain=true] [2025/03/13 17:00:57.304 +08:00] [INFO] [msgdispatcher/dispatcher.go:212] ["handle signal done"] [pchannel=by-dev-replicate-msg] [signal=start] [isMain=true] [2025/03/13 17:00:57.304 +08:00] [INFO] [msgdispatcher/client.go:114] ["register done"] [role=cdc-fd31c6b2e190440fb80a9683b6a83aa3] [nodeID=8444] [vchannel=by-dev-replicate-msg_847723f393fa4602893831c5001a006ev0] [dur=131.190393ms] [2025/03/13 17:00:57.304 +08:00] [INFO] [msgdispatcher/dispatcher.go:217] ["begin to work"] [pchannel=by-dev-replicate-msg] [isMain=true] [2025/03/13 17:00:57.322 +08:00] [INFO] [reader/etcd_op.go:566] ["get all collection data"] [count=2] [2025/03/13 17:00:57.323 +08:00] [INFO] [reader/replicate_channel_manager.go:167] ["has added dropped collection"] [ids="[]"] [2025/03/13 17:00:57.323 +08:00] [INFO] [reader/collection_reader.go:268] ["exist collection"] [task_id=847723f393fa4602893831c5001a006e] [name=t1] [collection_id=456616100077084995] [state=CollectionCreated] [seek_channels="[by-dev-rootcoord-dml_1]"] [2025/03/13 17:00:57.342 +08:00] [INFO] [reader/replicate_channel_manager.go:269] ["success to get the collection info in the target instance"] [collection_name=t1] [2025/03/13 17:00:57.342 +08:00] [INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true] [2025/03/13 17:00:57.371 +08:00] [INFO] [msgstream/mq_msgstream.go:214] ["Successfully create consumer"] [channel=by-dev-rootcoord-dml_1] [subname=by-dev-rootcoord-dml_18686642697158288638] [2025/03/13 17:00:57.371 +08:00] [INFO] [msgstream/mq_msgstream.go:535] ["MsgStream seek begin"] [channel=by-dev-rootcoord-dml_1] [MessageID=CA0QlQoYACAA] [includeCurrentMsg=false] [2025/03/13 17:00:57.482 +08:00] [INFO] [msgstream/mq_msgstream.go:541] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_1] [2025/03/13 17:00:57.482 +08:00] [INFO] [reader/stream_creator.go:135] ["success to seek the msg stream"] [channel_name=by-dev-rootcoord-dml_1_456616100077084995v0] [2025/03/13 17:00:57.482 +08:00] [INFO] [msgstream/mq_msgstream.go:224] ["start to close mq msg stream"] ["producer num"=0] ["consumer num"=1] [2025/03/13 17:00:57.489 +08:00] [INFO] [reader/replicate_channel_manager.go:728] ["create a replicate handler"] [source_channel=by-dev-rootcoord-dml_1] [target_channel=by-dev-rootcoord-dml_0] [2025/03/13 17:00:57.490 +08:00] [INFO] [reader/replicate_channel_manager.go:427] ["start read channel in the manager"] [nil_handler=false] [channel=by-dev-rootcoord-dml_1] [target_channel=by-dev-rootcoord-dml_0] [collection_id=456616100077084995] [2025/03/13 17:00:57.490 +08:00] [INFO] [reader/replicate_channel_manager.go:1275] ["start read channel in the handler"] [channel_name=by-dev-rootcoord-dml_1] [target_channel=by-dev-rootcoord-dml_0] [2025/03/13 17:00:57.490 +08:00] [INFO] [reader/replicate_channel_manager.go:438] ["start read the source channel"] [channel_name=by-dev-rootcoord-dml_1] [2025/03/13 17:00:57.490 +08:00] [INFO] [msgdispatcher/manager.go:67] ["create new dispatcherManager"] [role=cdc-a0b475adaa9540b5ba48264e8008db46] [nodeID=8444] [pchannel=by-dev-rootcoord-dml_1] [2025/03/13 17:00:57.490 +08:00] [INFO] [msgdispatcher/dispatcher.go:96] ["creating dispatcher..."] [pchannel=by-dev-rootcoord-dml_1] [subName=cdc-a0b475adaa9540b5ba48264e8008db46-8444-by-dev-rootcoord-dml_1_456616100077084995v0-true] [isMain=true] [2025/03/13 17:00:57.490 +08:00] [INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true] [2025/03/13 17:00:57.491 +08:00] [INFO] [msgdispatcher/manager.go:178] ["dispatcherManager is running..."] [role=cdc-a0b475adaa9540b5ba48264e8008db46] [nodeID=8444] [pchannel=by-dev-rootcoord-dml_1] [2025/03/13 17:00:57.493 +08:00] [INFO] [reader/etcd_op.go:710] ["get all partition data"] [partition_num=2] [2025/03/13 17:00:57.493 +08:00] [INFO] [reader/etcd_op.go:742] ["partition state is not created/dropped or partition name is default"] [partition_name=_default] [state=PartitionCreated] [2025/03/13 17:00:57.493 +08:00] [INFO] [reader/collection_reader.go:334] ["has started to read collection and partition"] [task_id=847723f393fa4602893831c5001a006e] [2025/03/13 17:00:57.511 +08:00] [INFO] [msgstream/mq_msgstream.go:929] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_1] [MessageID=CA0QlQoYACAA] [2025/03/13 17:00:57.634 +08:00] [INFO] [msgstream/mq_msgstream.go:939] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_1] [2025/03/13 17:00:57.664 +08:00] [INFO] [msgstream/mq_msgstream.go:1016] ["skip msg"] [type=TimeTick] [size=17] [msgTs=456617134898806785] [posTs=456617134899068928] [2025/03/13 17:00:57.664 +08:00] [INFO] [msgdispatcher/dispatcher.go:125] ["seek successfully"] [pchannel=by-dev-rootcoord-dml_1] [subName=cdc-a0b475adaa9540b5ba48264e8008db46-8444-by-dev-rootcoord-dml_1_456616100077084995v0-true] [isMain=true] [posTs=456617134899068928] [posTime=2025/03/13 16:55:35.937 +08:00] [tsLag=5m21.727457325s] [2025/03/13 17:00:57.664 +08:00] [INFO] [msgdispatcher/dispatcher.go:161] ["add new target"] [vchannel=by-dev-rootcoord-dml_1_456616100077084995v0] [isMain=true] [2025/03/13 17:00:57.664 +08:00] [INFO] [msgdispatcher/manager.go:115] ["add main dispatcher"] [role=cdc-a0b475adaa9540b5ba48264e8008db46] [nodeID=8444] [vchannel=by-dev-rootcoord-dml_1_456616100077084995v0] [2025/03/13 17:00:57.664 +08:00] [INFO] [msgdispatcher/dispatcher.go:189] ["get signal"] [pchannel=by-dev-rootcoord-dml_1] [signal=start] [isMain=true] [2025/03/13 17:00:57.664 +08:00] [INFO] [msgdispatcher/dispatcher.go:212] ["handle signal done"] [pchannel=by-dev-rootcoord-dml_1] [signal=start] [isMain=true] [2025/03/13 17:00:57.664 +08:00] [INFO] [msgdispatcher/client.go:114] ["register done"] [role=cdc-a0b475adaa9540b5ba48264e8008db46] [nodeID=8444] [vchannel=by-dev-rootcoord-dml_1_456616100077084995v0] [dur=173.996174ms] [2025/03/13 17:00:57.664 +08:00] [INFO] [reader/replicate_channel_manager.go:985] ["add collection to channel handler"] [channel_name=by-dev-rootcoord-dml_1_456616100077084995v0] [collection_id=456616100077084995] [collection_name=t1] [seek_channel=by-dev-rootcoord-dml_1] [2025/03/13 17:00:57.664 +08:00] [INFO] [reader/replicate_channel_manager.go:968] ["start to handle the msg pack"] [channel_name=by-dev-rootcoord-dml_1_456616100077084995v0] [2025/03/13 17:00:57.664 +08:00] [INFO] [msgdispatcher/dispatcher.go:217] ["begin to work"] [pchannel=by-dev-rootcoord-dml_1] [isMain=true] [2025/03/13 17:01:06.090 +08:00] [WARN] [msgdispatcher/dispatcher.go:255] ["lag target notified"] [pchannel=by-dev-rootcoord-dml_1] [isMain=true] [error="send target timeout, vchannel=by-dev-rootcoord-dml_1_456616100077084995v0, timeout=5s"] [2025/03/13 17:01:06.090 +08:00] [INFO] [msgdispatcher/manager.go:249] ["start splitting..."] [role=cdc-a0b475adaa9540b5ba48264e8008db46] [nodeID=8444] [vchannel=by-dev-rootcoord-dml_1_456616100077084995v0] [2025/03/13 17:01:06.090 +08:00] [INFO] [msgdispatcher/dispatcher.go:96] ["creating dispatcher..."] [pchannel=by-dev-rootcoord-dml_1] [subName=cdc-a0b475adaa9540b5ba48264e8008db46-8444-by-dev-rootcoord-dml_1_456616100077084995v0-false] [isMain=false] [2025/03/13 17:01:06.091 +08:00] [INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true] [2025/03/13 17:01:06.113 +08:00] [INFO] [msgstream/mq_msgstream.go:929] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_1] [MessageID=CA0Q7RYYACAA] [2025/03/13 17:01:06.219 +08:00] [INFO] [msgstream/mq_msgstream.go:939] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_1] [2025/03/13 17:01:06.230 +08:00] [INFO] [msgdispatcher/dispatcher.go:125] ["seek successfully"] [pchannel=by-dev-rootcoord-dml_1] [subName=cdc-a0b475adaa9540b5ba48264e8008db46-8444-by-dev-rootcoord-dml_1_456616100077084995v0-false] [isMain=false] [posTs=456617220042915841] [posTime=2025/03/13 17:01:00.735 +08:00] [tsLag=5.495625327s] [2025/03/13 17:01:06.230 +08:00] [INFO] [msgdispatcher/dispatcher.go:161] ["add new target"] [vchannel=by-dev-rootcoord-dml_1_456616100077084995v0] [isMain=false] [2025/03/13 17:01:06.230 +08:00] [INFO] [msgdispatcher/dispatcher.go:189] ["get signal"] [pchannel=by-dev-rootcoord-dml_1] [signal=start] [isMain=false] [2025/03/13 17:01:06.230 +08:00] [INFO] [msgdispatcher/dispatcher.go:212] ["handle signal done"] [pchannel=by-dev-rootcoord-dml_1] [signal=start] [isMain=false] [2025/03/13 17:01:06.230 +08:00] [INFO] [msgdispatcher/manager.go:271] ["split done"] [role=cdc-a0b475adaa9540b5ba48264e8008db46] [nodeID=8444] [vchannel=by-dev-rootcoord-dml_1_456616100077084995v0] [2025/03/13 17:01:06.230 +08:00] [INFO] [msgdispatcher/dispatcher.go:217] ["begin to work"] [pchannel=by-dev-rootcoord-dml_1] [isMain=false]

huangping11 avatar Mar 13 '25 09:03 huangping11

@huangping11 Yes, you need to change ttMsgEnable of target milvus from false to true

SimFG avatar Mar 13 '25 11:03 SimFG

@SimFG Do I need to restart Milvus after the modification, and can it take effect online?

huangping11 avatar Mar 14 '25 06:03 huangping11

Yes, but it will take a few seconds to take effect.

SimFG avatar Mar 14 '25 07:03 SimFG

@SimFG Do you mean that there is no need to restart after the modification, and it will automatically return to normal in a few minutes?

huangping11 avatar Mar 14 '25 09:03 huangping11

@SimFG If it is a database that has been running for a period of time, can it be restored using a backup and then perform incremental synchronization?

huangping11 avatar Mar 14 '25 09:03 huangping11

@SimFG I directly copied the volumes directory of the source database to the target environment. After starting the synchronization, I found that the data that had already been loaded and then deleted would not be synchronized to the target environment.

huangping11 avatar Mar 14 '25 09:03 huangping11

@SimFG If it is a database that has been running for a period of time, can it be restored using a backup and then perform incremental synchronization?

yes, it can

SimFG avatar Mar 14 '25 10:03 SimFG

@SimFG I directly copied the volumes directory of the source database to the target environment. After starting the synchronization, I found that the data that had already been loaded and then deleted would not be synchronized to the target environment.

This will not work because the files will not be recognized by the new cluster, such as the collection id etc.

SimFG avatar Mar 14 '25 10:03 SimFG

@SimFG If the source environment is already in the running stage (with data), how can I use Milvus CDC to synchronize it to the target environment? Should we use milvus-backup to back up and restore to the target environment, and then perform incremental synchronization?

huangping11 avatar Mar 17 '25 03:03 huangping11

@SimFG If it is possible to use the backup and restore method to build the Milvus-CDC synchronization, is the source environment allowed to write new data during the backup and restore process? If new data is written, will it lead to inconsistencies between the source environment and the target environment?

huangping11 avatar Mar 17 '25 03:03 huangping11

@SimFG I found that changing the ttMsgEnable of the target Milvus from false to true won't take effect. It requires restarting the milvus-standalone to make it work.

huangping11 avatar Mar 17 '25 08:03 huangping11

How do you determine that the ttMsgEnable configuration does not take effect?

SimFG avatar Mar 17 '25 11:03 SimFG

Do you have a need for active/standby disaster recovery? If so, you can communicate with our sales team. This capability cannot be achieved by CDC alone, and involves many details. At the same time, this part is also under development and testing, and there may be major changes in the future. Recently, our cloud will launch this function for beta testing.

If it is for personal use, I recommend stopping the service and using CDC. If it is enterprise-level, you can wait until it is launched on the cloud and try it out.

SimFG avatar Mar 17 '25 11:03 SimFG