starrocks [BugFix] Fix streamLoad import large number of columns failed with er…

trafficstars

What type of PR is this：

[x] bug
[ ] feature
[ ] enhancement
[ ] refactor
[ ] others

Which issues of this PR fixes ：

Fixes #9865

Problem Summary(Required) ：

Fe's HTTP service is implemented by netty, netty HttpServerCodec has default properties： http_max_initial_line_length=4096 http_max_header_size=8192 http_max_chunk_size=8192

when use streamLoad import large number of columns, http header properties "columns" will be more than http_max_header_size,but netty decode not throw exception and return an empty header request. Next, the auth information will not be found in the Request execution phase. So we need to validate the request before executing it and return a true error message.

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] I have added user document for my new feature or new function

Aug 11 '22 03:08 srwangit

All committers have signed the CLA.

Aug 11 '22 03:08 CLAassistant

run starrocks_fe_unittest

Aug 12 '22 07:08 meegoo

thanks your contribution. After increase http_max_header_size, does sr load success when import with large number of columns? I'm not sure because FE will redirect request to BE, does it works well too?

Aug 12 '22 07:08 meegoo

[FE PR Coverage Check]

:heart_eyes: pass : 18 / 22 (81.82%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
:large_blue_circle:	com/starrocks/http/HttpRequestException.java	2	4	50.00%	[11, 12]
:large_blue_circle:	com/starrocks/http/HttpServerHandler.java	12	14	85.71%	[65, 78]
:large_blue_circle:	com/starrocks/common/Config.java	3	3	100.00%	[]
:large_blue_circle:	com/starrocks/http/HttpServer.java	1	1	100.00%	[]

Aug 12 '22 07:08 wanpengfei-git

thanks your contribution. After increase http_max_header_size, does sr load success when import with large number of columns? I'm not sure because FE will redirect request to BE, does it works well too?

@meegoo I've successfully executed Streamload with headers over 10000. I'm not sure what the upper limit of HTTP requests for BE. If redirect to BE execute fails, it should be the BE's fault.

The main purpose of this PR is to return the true cause of the failure and set the appropriate values through the configuration file. The default values are now the same as before, i think this has no effect on Fe.

Aug 12 '22 07:08 srwangit

SonarCloud Quality Gate failed.