enhance: add disk file writer with Direct IO support

[sparknack]

This patch introduces a disk file writer that supports Direct IO.

Currently, it is exclusively utilized during the QueryNode load process.

Below is its parameters:

1. common.diskWriteMode

• Function: Determines the disk write mode.
• Optional Values: direct: Enables Direct IO, bypassing the system page cache for direct disk access. This is suitable for scenarios requiring high I/O performance and precise control over data writes. buffered (default): Uses buffered I/O, where data is first written to the system page cache and then flushed to the disk.

2. common.diskWriteBufferSizeKb

• Function: Configures the buffer size for Direct IO writes (ignored in buffered mode). This configuration can be used to adjust the number of write IOs and control the disk write bandwidth in conjunction with other configuration like cgroup.
• Valid Range: 4 to 2048 (in kilobytes).
• Key Requirement: The value must be aligned to 4KB.
• Default: 64 KB.

Both parameters can be updated during runtime.

[mergify[bot]]

@sparknack

Invalid PR Title Format Detected

Your PR submission does not adhere to our required standards. To ensure clarity and consistency, please meet the following criteria:

1. Title Format: The PR title must begin with one of these prefixes:

• feat: for introducing a new feature.
• fix: for bug fixes.
• enhance: for improvements to existing functionality.
• test: for add tests to existing functionality.
• doc: for modifying documentation.
• auto: for the pull request from bot.

2. Description Requirement: The PR must include a non-empty description, detailing the changes and their impact.

Required Title Structure:

[Type]: [Description of the PR]

Where Type is one of feat, fix, enhance, test or doc.

Example:

enhance: improve search performance significantly 

Please review and update your PR to comply with these guidelines.

[mergify[bot]]

@sparknack go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@sparknack E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@sparknack cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

[mergify[bot]]

@sparknack E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@sparknack cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

[mergify[bot]]

@sparknack go-sdk check failed, comment rerun go-sdk can trigger the job again.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 85.45035% with 63 lines in your changes missing coverage. Please review.

Project coverage is 78.88%. Comparing base (26ec841) to head (4cb89e0). Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/core/src/storage/FileWriter.cpp	85.14%	26 Missing :warning:
internal/core/src/storage/storage_c.cpp	0.00%	14 Missing :warning:
internal/querynodev2/server.go	38.09%	12 Missing and 1 partial :warning:
internal/core/src/common/File.h	60.00%	2 Missing :warning:
internal/core/src/storage/DiskFileManagerImpl.cpp	90.47%	2 Missing :warning:
internal/core/src/storage/FileWriter.h	94.59%	2 Missing :warning:
internal/core/src/common/ChunkWriter.cpp	98.11%	1 Missing :warning:
internal/core/src/common/ChunkWriter.h	88.88%	1 Missing :warning:
internal/core/src/index/VectorMemIndex.cpp	90.90%	1 Missing :warning:
...toragev1translator/DefaultValueChunkTranslator.cpp	0.00%	1 Missing :warning:

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #42665       +/-   ##
===========================================
+ Coverage   73.78%   78.88%    +5.10%     
===========================================
  Files         342     1559     +1217     
  Lines       32023   222269   +190246     
===========================================
+ Hits        23627   175330   +151703     
- Misses       8396    40462    +32066     
- Partials        0     6477     +6477     

Components	Coverage Δ
Client	79.39% <ø> (∅)
Core	73.87% <86.26%> (+0.09%)	:arrow_up:
Go	79.83% <81.15%> (∅)

Files with missing lines	Coverage Δ
internal/core/src/common/ChunkTarget.cpp	100.00% <100.00%> (+26.31%)	:arrow_up:
internal/core/src/common/ChunkTarget.h	100.00% <100.00%> (ø)
internal/core/src/index/BitmapIndex.cpp	83.68% <100.00%> (+0.34%)	:arrow_up:
internal/core/src/index/BitmapIndex.h	56.75% <ø> (ø)
internal/core/src/index/HybridScalarIndex.h	37.20% <ø> (ø)
internal/core/src/index/StringIndexMarisa.cpp	75.24% <100.00%> (+0.97%)	:arrow_up:
internal/core/src/monitor/prometheus_client.h	100.00% <ø> (ø)
...rnal/core/src/segcore/ChunkedSegmentSealedImpl.cpp	81.47% <100.00%> (-0.02%)	:arrow_down:
...ternal/core/src/segcore/ChunkedSegmentSealedImpl.h	54.76% <ø> (ø)
internal/core/src/segcore/Utils.cpp	73.58% <ø> (ø)
... and 16 more

... and 1214 files with indirect coverage changes

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mergify[bot]]

@sparknack cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

[sparknack]

rerun cpp-unit-test

[mergify[bot]]

@sparknack cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

[mergify[bot]]

@sparknack cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

[mergify[bot]]

@sparknack cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

[sparknack]

rerun cpp-unit-test

[mergify[bot]]

@sparknack Please associate the related issue to the body of your Pull Request. (eg. "issue: #")

[mergify[bot]]

@sparknack cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

[sparknack]

rerun cpp-unit-test

[sparknack]

/hold

[mergify[bot]]

@sparknack cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@sparknack cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@sparknack go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@sparknack cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@sparknack cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@sparknack go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@sparknack cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

[sparknack]

/unhold

[zhengbuqian]

/approve

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sparknack, zhengbuqian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~OWNERS~~ [zhengbuqian]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment