milvus icon indicating copy to clipboard operation
milvus copied to clipboard

Remove datanode timetick mq, use rpc to report instead

Open wayblink opened this issue 1 year ago • 39 comments

Remove timetick mq logic in datacoord and datanode, use RPC instead.

datacoord: 1, Remove DataNodeTtLoop and related msgstream consumer 2, Add ReportDataNodeTtMsgs interface, ReportDataNodeTtMsgs call handleTimetickMessage

datanode: 1, Remove related msgstream producer in flow graph 2, Introduce a timetick_sender in datanode. All insert_buffer_nodes on datanode updates channel states to timetick_sender. 3, timetick_sender call datacoord.ReportDataNodeTtMsgs to send DataNodeTTMsg periodically, perform like heartbeat

related: #22953

wayblink avatar Mar 31 '23 06:03 wayblink

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar Mar 31 '23 06:03 mergify[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Mar 31 '23 06:03 mergify[bot]

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar Mar 31 '23 06:03 mergify[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Mar 31 '23 07:03 mergify[bot]

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar Mar 31 '23 07:03 mergify[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Mar 31 '23 08:03 mergify[bot]

/run-cpu-e2e

wayblink avatar Mar 31 '23 08:03 wayblink

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar Mar 31 '23 09:03 mergify[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Mar 31 '23 09:03 mergify[bot]

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar Mar 31 '23 10:03 mergify[bot]

Codecov Report

Merging #23156 (31cf5c9) into master (4387f36) will decrease coverage by 0.33%. The diff coverage is 82.70%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #23156      +/-   ##
==========================================
- Coverage   82.44%   82.11%   -0.33%     
==========================================
  Files         779      780       +1     
  Lines      102453   102681     +228     
==========================================
- Hits        84466    84320     -146     
- Misses      14972    15316     +344     
- Partials     3015     3045      +30     
Impacted Files Coverage Δ
internal/util/mock/grpc_datacoord_client.go 89.01% <0.00%> (-2.01%) :arrow_down:
internal/datacoord/server.go 62.71% <20.00%> (-11.15%) :arrow_down:
internal/distributed/datacoord/client/client.go 92.61% <40.00%> (-2.71%) :arrow_down:
internal/datacoord/services.go 85.15% <67.92%> (-0.68%) :arrow_down:
internal/datanode/flow_graph_insert_buffer_node.go 78.81% <80.00%> (-15.13%) :arrow_down:
internal/datanode/timetick_sender.go 95.65% <95.65%> (ø)
internal/datanode/data_node.go 73.74% <100.00%> (+0.01%) :arrow_up:
internal/datanode/data_sync_service.go 78.78% <100.00%> (-1.90%) :arrow_down:
internal/datanode/flow_graph_manager.go 95.17% <100.00%> (-4.83%) :arrow_down:
internal/distributed/datacoord/service.go 92.30% <100.00%> (+0.07%) :arrow_up:
... and 1 more

... and 23 files with indirect coverage changes

codecov[bot] avatar Mar 31 '23 11:03 codecov[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Mar 31 '23 11:03 mergify[bot]

/run-cpu-e2e

wayblink avatar Mar 31 '23 11:03 wayblink

/assign @XuanYang-cn

wayblink avatar Mar 31 '23 11:03 wayblink

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Mar 31 '23 12:03 mergify[bot]

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar Apr 06 '23 03:04 mergify[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Apr 06 '23 04:04 mergify[bot]

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar Apr 06 '23 07:04 mergify[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Apr 06 '23 07:04 mergify[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Apr 10 '23 02:04 mergify[bot]

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar Apr 10 '23 02:04 mergify[bot]

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar Apr 10 '23 03:04 mergify[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Apr 10 '23 03:04 mergify[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Apr 10 '23 03:04 mergify[bot]

/run-cpu-e2e

wayblink avatar Apr 10 '23 06:04 wayblink

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Apr 10 '23 07:04 mergify[bot]

/run-cpu-e2e

wayblink avatar Apr 10 '23 08:04 wayblink

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Apr 10 '23 08:04 mergify[bot]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Apr 10 '23 13:04 mergify[bot]

@wayblink ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar Apr 10 '23 13:04 mergify[bot]