starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY

Open rickif opened this issue 1 year ago • 4 comments

Why I'm doing:

Now, the SR does't support loading JSON to complex type column.

What I'm doing:

This PR adds the support of JSON format loading to STRUCT/MAP/ARRAY type column.

Here are some examples.

create table tbl_map (key1 int, key2 map<string,string>);

curl --location-trusted -u root:  'http://127.0.0.1:18040/api/db0/tbl_map/_stream_load'
 \-X PUT  \-H 'format: json' -H "columns:key1,key2" 
-d '{"key1":1,"key2":{"key3":"aaa"}}'
create tablet tbl_struct (key1 int, key2 struct<key3, string>);

curl --location-trusted -u root:  'http://127.0.0.1:18040/api/db0/tbl_struct/_stream_load'
 \-X PUT  \-H 'format: json' -H "columns:key1,key2" 
-d '{"key1":1,"key2":{"key3":"aaa"}}'
create tablet tbl_array (key1 int, key2 array<string>);

curl --location-trusted -u root:  'http://127.0.0.1:18040/api/db0/tbl_array/_stream_load' 
\-X PUT  \-H 'format: json' -H "columns:key1,key2" 
-d '{"key1":1,"key2":["aaa","bbb"]}'

Fixes https://github.com/StarRocks/starrocks/issues/43101

What type of PR is this:

  • [ ] BugFix
  • [x] Feature
  • [ ] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [x] Yes, this PR will result in a change in behavior.
  • [ ] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [x] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [x] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [ ] 3.3
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

rickif avatar May 10 '24 04:05 rickif

@mergifyio rebase

rickif avatar May 12 '24 06:05 rickif

rebase

✅ Branch has been successfully rebased

mergify[bot] avatar May 12 '24 06:05 mergify[bot]

@mergifyio rebase

rickif avatar May 28 '24 23:05 rickif

rebase

✅ Branch has been successfully rebased

mergify[bot] avatar May 28 '24 23:05 mergify[bot]

[FE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar May 29 '24 04:05 github-actions[bot]

[BE Incremental Coverage Report]

:white_check_mark: pass : 118 / 134 (88.06%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: be/src/formats/json/nullable_column.cpp 38 46 82.61% [213, 214, 234, 235, 249, 250, 267, 268]
:large_blue_circle: be/src/formats/json/struct_column.cpp 16 19 84.21% [29, 30, 31]
:large_blue_circle: be/src/formats/json/map_column.cpp 25 28 89.29% [30, 31, 32]
:large_blue_circle: be/src/exec/json_scanner.cpp 39 41 95.12% [279, 550]

github-actions[bot] avatar May 29 '24 04:05 github-actions[bot]

https://github.com/Mergifyio backport branch-3.3

wyb avatar May 30 '24 05:05 wyb

backport branch-3.3

✅ Backports have been created

mergify[bot] avatar May 30 '24 05:05 mergify[bot]

https://github.com/Mergifyio backport branch-3.2

wyb avatar Jun 20 '24 01:06 wyb

backport branch-3.2

✅ Backports have been created

mergify[bot] avatar Jun 20 '24 01:06 mergify[bot]