milvus icon indicating copy to clipboard operation
milvus copied to clipboard

feat: impl StructArray -- data storage support in go

Open SpadeA-Tang opened this issue 6 months ago • 30 comments

Ref https://github.com/milvus-io/milvus/issues/42148

This PR enables go part storage to support array of vector type.

SpadeA-Tang avatar May 30 '25 09:05 SpadeA-Tang

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: SpadeA-Tang To complete the pull request process, please assign tedxu after the PR has been reviewed. You can assign the PR to them by writing /assign @tedxu in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot avatar May 30 '25 09:05 sre-ci-robot

@SpadeA-Tang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar May 30 '25 10:05 mergify[bot]

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar May 30 '25 10:05 mergify[bot]

Codecov Report

Attention: Patch coverage is 56.09137% with 173 lines in your changes missing coverage. Please review.

Project coverage is 80.48%. Comparing base (d35c33d) to head (3220abe). Report is 28 commits behind head on master.

Files with missing lines Patch % Lines
internal/storage/insert_data.go 51.13% 42 Missing and 1 partial :warning:
internal/storage/utils.go 33.89% 36 Missing and 3 partials :warning:
internal/storage/data_codec.go 55.00% 25 Missing and 2 partials :warning:
internal/storage/serde_events.go 67.08% 19 Missing and 7 partials :warning:
internal/storage/serde.go 38.09% 10 Missing and 3 partials :warning:
internal/storage/payload_writer.go 60.86% 8 Missing and 1 partial :warning:
internal/storage/payload_reader.go 62.50% 4 Missing and 2 partials :warning:
internal/storage/schema.go 78.94% 3 Missing and 1 partial :warning:
internal/storage/arrow_util.go 62.50% 2 Missing and 1 partial :warning:
internal/storage/serde_events_v2.go 60.00% 2 Missing :warning:
... and 1 more

:x: Your patch status has failed because the patch coverage (56.09%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #42453      +/-   ##
==========================================
- Coverage   81.81%   80.48%   -1.33%     
==========================================
  Files        1211     1552     +341     
  Lines      189086   221307   +32221     
==========================================
+ Hits       154704   178128   +23424     
- Misses      28001    36777    +8776     
- Partials     6381     6402      +21     
Components Coverage Δ
Client 79.47% <ø> (ø)
Core 72.78% <ø> (∅)
Go 82.01% <56.09%> (-0.04%) :arrow_down:
Files with missing lines Coverage Δ
...nternal/datanode/compactor/clustering_compactor.go 64.86% <100.00%> (ø)
internal/datanode/compactor/segment_writer.go 64.77% <100.00%> (-0.90%) :arrow_down:
internal/storage/data_sorter.go 90.00% <100.00%> (+0.89%) :arrow_up:
internal/util/importutilv2/binlog/reader.go 55.37% <100.00%> (ø)
internal/flushcommon/syncmgr/pack_writer_v2.go 76.66% <50.00%> (ø)
internal/storage/serde_events_v2.go 75.13% <60.00%> (+1.60%) :arrow_up:
internal/storage/arrow_util.go 53.44% <62.50%> (-0.28%) :arrow_down:
internal/storage/schema.go 84.61% <78.94%> (-3.39%) :arrow_down:
internal/storage/payload_reader.go 68.29% <62.50%> (-0.16%) :arrow_down:
internal/storage/payload_writer.go 89.75% <60.86%> (-1.07%) :arrow_down:
... and 5 more

... and 380 files with indirect coverage changes

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar May 30 '25 12:05 codecov[bot]

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 03 '25 03:06 mergify[bot]

@SpadeA-Tang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Jun 03 '25 06:06 mergify[bot]

@SpadeA-Tang let me know if the PR is ready to be reviewed.

xiaofan-luan avatar Jun 03 '25 15:06 xiaofan-luan

@SpadeA-Tang let me know if the PR is ready to be reviewed.

Yes

SpadeA-Tang avatar Jun 03 '25 15:06 SpadeA-Tang

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 04 '25 08:06 mergify[bot]

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 04 '25 10:06 mergify[bot]

@SpadeA-Tang cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify[bot] avatar Jun 04 '25 11:06 mergify[bot]

@SpadeA-Tang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Jun 04 '25 12:06 mergify[bot]

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 11 '25 10:06 mergify[bot]

rerun go-sdk

SpadeA-Tang avatar Jun 11 '25 11:06 SpadeA-Tang

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 11 '25 13:06 mergify[bot]

@SpadeA-Tang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Jun 11 '25 15:06 mergify[bot]

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 12 '25 13:06 mergify[bot]

@SpadeA-Tang cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify[bot] avatar Jun 12 '25 17:06 mergify[bot]

@SpadeA-Tang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Jun 12 '25 18:06 mergify[bot]

rerun go-sdk

SpadeA-Tang avatar Jun 13 '25 02:06 SpadeA-Tang

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 13 '25 03:06 mergify[bot]

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 13 '25 09:06 mergify[bot]

rerun go-sdk

SpadeA-Tang avatar Jun 13 '25 09:06 SpadeA-Tang

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 13 '25 09:06 mergify[bot]

@SpadeA-Tang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Jun 13 '25 09:06 mergify[bot]

rerun go-sdk

SpadeA-Tang avatar Jun 13 '25 09:06 SpadeA-Tang

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 13 '25 10:06 mergify[bot]

@SpadeA-Tang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Jun 13 '25 10:06 mergify[bot]

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Jun 16 '25 03:06 mergify[bot]

@SpadeA-Tang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Jun 16 '25 03:06 mergify[bot]