Use log id instead of log path of the binlog before persisting

[jaime0815]

/kind improvement Signed-off-by: yun.zhang [email protected]

[mergify[bot]]

@jaime0815 ut workflow job failed, comment rerun ut can trigger the job again.

[codecov[bot]]

Codecov Report

Merging #19123 (fd94f0d) into master (2cfacbb) will increase coverage by 0.02%. The diff coverage is 87.79%.

@@            Coverage Diff             @@
##           master   #19123      +/-   ##
==========================================
+ Coverage   80.47%   80.49%   +0.02%     
==========================================
  Files         675      676       +1     
  Lines       94516    94616     +100     
==========================================
+ Hits        76064    76164     +100     
+ Misses      15221    15220       -1     
- Partials     3231     3232       +1     

Impacted Files	Coverage Δ
internal/datacoord/server.go	76.80% <69.23%> (ø)
internal/metastore/kv/datacoord/kv_catalog.go	83.81% <84.53%> (+4.58%)	:arrow_up:
internal/datacoord/meta.go	87.98% <87.50%> (+0.03%)	:arrow_up:
internal/util/etcd/etcd_util.go	87.75% <93.87%> (+6.12%)	:arrow_up:
internal/datanode/allocator.go	100.00% <100.00%> (ø)
internal/datanode/binlog_io.go	95.26% <100.00%> (ø)
internal/datanode/data_node.go	82.27% <100.00%> (ø)
internal/datanode/flush_manager.go	93.78% <100.00%> (ø)
internal/metastore/kv/rootcoord/kv_catalog.go	86.32% <100.00%> (-0.18%)	:arrow_down:
internal/querycoordv2/handlers.go	73.12% <100.00%> (+1.23%)	:arrow_up:
... and 19 more

[mergify[bot]]

@jaime0815 ut workflow job failed, comment rerun ut can trigger the job again.

[jaime0815]

rerun ut

[mergify[bot]]

@jaime0815 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[xiaofan-luan]

One thing we could do is just to clean up all segment meta into new format when datacoord upgrade to new version, but it definitely take a very long time to do the transformation, the code itself could be much clean

[mergify[bot]]

@jaime0815 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@jaime0815 ut workflow job failed, comment rerun ut can trigger the job again.

[jaime0815]

One thing we could do is just to clean up all segment meta into new format when datacoord upgrade to new version, but it definitely take a very long time to do the transformation, the code itself could be much clean

old format segment meta will not clean up when upgrading, as you said the disadvantage is uncontrollable. now it converts to the new format when compacting, but this solution also has a drawback that brings some unnecessary update operations on the etcd,

[mergify[bot]]

@jaime0815 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[jaime0815]

/run-cpu-e2e

[mergify[bot]]

@jaime0815 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@jaime0815 ut workflow job failed, comment rerun ut can trigger the job again.

[mergify[bot]]

@jaime0815 ut workflow job failed, comment rerun ut can trigger the job again.

[mergify[bot]]

@jaime0815 ut workflow job failed, comment rerun ut can trigger the job again.

[mergify[bot]]

@jaime0815 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@jaime0815 ut workflow job failed, comment rerun ut can trigger the job again.

[mergify[bot]]

@jaime0815 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@jaime0815 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@jaime0815 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@jaime0815 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@jaime0815 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[xiaofan-luan]

/lgtm

[xiaofan-luan]

/approve

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jaime0815, xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~OWNERS~~ [xiaofan-luan]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[xiaofan-luan]

/lgtm