milvus icon indicating copy to clipboard operation
milvus copied to clipboard

enhance: add ScalarFieldProto& overload to avoid unnecessary copies

Open zhengbuqian opened this issue 1 week ago • 4 comments

  1. Array.h: Add output_data(ScalarFieldProto&) overload for both Array and ArrayView classes
  2. Use std::string_view instead of std::string for VARCHAR and GEOMETRY types to avoid extra string copies
  3. Call Reserve(length_) before writing to proto objects to reduce memory reallocations

a simple test shows those optimizations improve the Array of Varchar bulk_subscript performance by 20%

issue: https://github.com/milvus-io/milvus/issues/45679

zhengbuqian avatar Nov 20 '25 14:11 zhengbuqian

[ci-v2-notice] Notice: We are gradually rolling out the new ci-v2 system.

  • Legacy CI jobs remain unaffected, you can just ignore ci-v2 if you don't want to run it.
  • Additional "ci-v2/*" checkers will run for this PR to ensure the new ci-v2 system is working as expected.
  • For tests that exist in both v1 and v2, passing in either system is considered PASS.

To rerun ci-v2 checks, comment with:

  • /ci-rerun-code-check // for ci-v2/code-check
  • /ci-rerun-build // for ci-v2/build
  • /ci-rerun-ut-integration // for ci-v2/ut-integration
  • /ci-rerun-ut-go // for ci-v2/ut-go
  • /ci-rerun-ut-cpp // for ci-v2/ut-cpp
  • /ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp
  • /ci-rerun-e2e-arm // for ci-v2/e2e-arm [master branch only]
  • /ci-rerun-e2e-default // for ci-v2/e2e-default [master branch only]

If you have any questions or requests, please contact @zhikunyao.

sre-ci-robot avatar Nov 20 '25 15:11 sre-ci-robot

@zhengbuqian cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Nov 20 '25 15:11 mergify[bot]

@zhengbuqian go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Nov 20 '25 15:11 mergify[bot]

/ci-rerun-ut

zhengbuqian avatar Nov 21 '25 02:11 zhengbuqian

rerun go-sdk

zhengbuqian avatar Nov 21 '25 02:11 zhengbuqian

/run-cpu-e2e

zhengbuqian avatar Nov 21 '25 02:11 zhengbuqian

@zhengbuqian go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Nov 21 '25 02:11 mergify[bot]

@zhengbuqian cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Nov 21 '25 02:11 mergify[bot]

Codecov Report

:x: Patch coverage is 77.55102% with 11 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 76.48%. Comparing base (8ee8c01) to head (1888370). :warning: Report is 22 commits behind head on master.

Files with missing lines Patch % Lines
internal/core/src/common/Array.h 80.48% 8 Missing :warning:
internal/core/src/mmap/ChunkedColumnGroup.h 0.00% 3 Missing :warning:
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #45743      +/-   ##
==========================================
- Coverage   76.51%   76.48%   -0.04%     
==========================================
  Files        1875     1875              
  Lines      292559   292602      +43     
==========================================
- Hits       223865   223782      -83     
- Misses      61271    61389     +118     
- Partials     7423     7431       +8     
Components Coverage Δ
Client 78.17% <ø> (ø)
Core 83.20% <77.55%> (+<0.01%) :arrow_up:
Go 74.58% <ø> (-0.07%) :arrow_down:
Files with missing lines Coverage Δ
internal/core/src/mmap/ChunkedColumn.h 74.36% <100.00%> (-0.17%) :arrow_down:
internal/core/src/mmap/ChunkedColumnInterface.h 64.70% <ø> (ø)
...rnal/core/src/segcore/ChunkedSegmentSealedImpl.cpp 64.75% <100.00%> (ø)
internal/core/src/mmap/ChunkedColumnGroup.h 44.79% <0.00%> (+0.28%) :arrow_up:
internal/core/src/common/Array.h 90.62% <80.48%> (-0.22%) :arrow_down:

... and 32 files with indirect coverage changes

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Nov 21 '25 06:11 codecov[bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liliu-z, zhengbuqian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot avatar Nov 21 '25 10:11 sre-ci-robot