[Feature] Bypass sdk read and write decoupling

[Jay-ju]

Why I'm doing:

Fixes #47367

What I'm doing:

You can learn more about the background from pr's https://github.com/StarRocks/starrocks/pull/38466.

This PR is mainly used to solve the problem that when BE directly reads and writes SR file format on S3, the code cannot be encapsulated into SDK form. This PR is mainly to decouple some code implementation and provide some compilation scripts.

build format lib

BUILD_TYPE=Debug ./build.sh --format-lib  -j32

will install directory:

-- Installing: /data00/code/community/starrocks/be/output/format-lib/
-- Installing: /data00/code/community/starrocks/be/output/format-lib/libstarrocks_format.so
Skip Building Java Extensions
***************************************
Successfully build StarRocks  √ Format Lib ; StartTime:2024-05-20 09:52:29, EndTime:2024-05-20 09:58:21, TotalTime:352s

What type of PR is this:

• [ ] BugFix
• [x] Feature
• [x] Enhancement
• [ ] Refactor
• [ ] UT
• [ ] Doc
• [ ] Tool

Does this PR entail a change in behavior?

• [ ] Yes, this PR will result in a change in behavior.
• [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• [x] Interface/UI changes: syntax, type conversion, expression evaluation, display information
• [ ] Parameter changes: default values, similar parameters but with different default values
• [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
• [ ] Feature removed
• [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• [x] I have added test cases for my bug fix or my new feature
• [ ] This pr needs user documentation (for new or modified features or behaviors)
  • [ ] I have added documentation for my new feature or new function
• [ ] This is a backport pr

Bugfix cherry-pick branch check:

• [x] I have checked the version labels which the pr will be auto-backported to the target branch
  • [x] 3.3
  • [ ] 3.2
  • [ ] 3.1
  • [ ] 3.0
  • [ ] 2.5

[sonarqubecloud[bot]]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[github-actions[bot]]

[Java-Extensions Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

[github-actions[bot]]

[FE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

[github-actions[bot]]

[BE Incremental Coverage Report]

:white_check_mark: pass : 262 / 314 (83.44%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
:large_blue_circle:	be/src/runtime/fragment_mgr.cpp	0	1	00.00%	[878]
:large_blue_circle:	be/src/runtime/descriptors.cpp	2	6	33.33%	[820, 822, 824, 827]
:large_blue_circle:	be/src/storage/lake/general_tablet_writer.cpp	12	22	54.55%	[74, 98, 99, 266, 267, 268, 270, 272, 293, 294]
:large_blue_circle:	be/src/storage/lake/pk_tablet_writer.cpp	4	6	66.67%	[67, 68]
:large_blue_circle:	be/src/fs/fs_s3.cpp	43	61	70.49%	[138, 303, 379, 382, 383, 393, 403, 408, 415, 525, 526, 527, 528, 529, 530, 555, 556, 557]
:large_blue_circle:	be/src/fs/fs.cpp	4	5	80.00%	[93]
:large_blue_circle:	be/src/storage/protobuf_file.cpp	17	20	85.00%	[60, 77, 155]
:large_blue_circle:	be/src/storage/lake/rowset.cpp	33	38	86.84%	[177, 438, 460, 461, 472]
:large_blue_circle:	be/src/storage/lake/lake_delvec_loader.cpp	14	16	87.50%	[28, 31]
:large_blue_circle:	be/src/storage/lake/tablet_manager.cpp	15	17	88.24%	[72, 73]
:large_blue_circle:	be/src/storage/lake/tablet.cpp	12	13	92.31%	[113]
:large_blue_circle:	be/src/io/s3_input_stream.cpp	42	45	93.33%	[70, 75, 111]
:large_blue_circle:	be/src/types/timestamp_value.cpp	7	7	100.00%	[]
:large_blue_circle:	be/src/storage/lake/meta_file.cpp	9	9	100.00%	[]
:large_blue_circle:	be/src/storage/lake/lake_delvec_loader.h	5	5	100.00%	[]
:large_blue_circle:	be/src/storage/rowset/indexed_column_reader.cpp	2	2	100.00%	[]
:large_blue_circle:	be/src/runtime/exec_env.cpp	2	2	100.00%	[]
:large_blue_circle:	be/src/storage/rowset/scalar_column_iterator.cpp	3	3	100.00%	[]
:large_blue_circle:	be/src/storage/lake/horizontal_compaction_task.cpp	2	2	100.00%	[]
:large_blue_circle:	be/src/storage/lake/vertical_compaction_task.cpp	2	2	100.00%	[]
:large_blue_circle:	be/src/storage/lake/tablet_reader.cpp	2	2	100.00%	[]
:large_blue_circle:	be/src/storage/lake/update_manager.cpp	9	9	100.00%	[]
:large_blue_circle:	be/src/storage/lake/tablet_manager.h	1	1	100.00%	[]
:large_blue_circle:	be/src/storage/rowset/segment.cpp	1	1	100.00%	[]
:large_blue_circle:	be/src/storage/lake/lake_primary_key_compaction_conflict_resolver.h	1	1	100.00%	[]
:large_blue_circle:	be/src/storage/protobuf_file.h	1	1	100.00%	[]
:large_blue_circle:	be/src/storage/lake/tablet.h	4	4	100.00%	[]
:large_blue_circle:	be/src/fs/fs.h	2	2	100.00%	[]
:large_blue_circle:	be/src/storage/lake/lake_primary_key_compaction_conflict_resolver.cpp	3	3	100.00%	[]
:large_blue_circle:	be/src/fs/hdfs/fs_hdfs.cpp	2	2	100.00%	[]
:large_blue_circle:	be/src/io/s3_input_stream.h	6	6	100.00%	[]

[Samrose-Ahmed]

I am trying to understand this PR, it seems to aim to achieve being able to directly read and write SR files. But what is the 'format-lib' that is being compiled and built, can you explain?

[kevincai]

@Samrose-Ahmed yes, this PR will provide the ability to build the library format-lib, and then other application can leverage the library to interact with Frontend and has the ability to write data to remote storage directly without proxying through backend RPC.

[Samrose-Ahmed]

Awesome thanks.

[wangsimo0]

ignore backport check: 3.3.5