starrocks
starrocks copied to clipboard
[Feature] support read map column from parquet file
support read map column from parquet file
Signed-off-by: zombee0 [email protected]
What type of PR is this:
- [ ] BugFix
- [x] Feature
- [ ] Enhancement
- [ ] Refactor
- [ ] UT
- [ ] Doc
- [ ] Tool
Which issues of this PR fixes :
Fixes #10677
Problem Summary(Required) :
there are two main change:
- generate the IORange for map type column, the map type in parquet is represented by MAP<Struct<Key,Value>>, so we should get IORange by
for (auto& field : schema_node->children[0].children) { int64_t range_end = 0; _collect_field_io_range(field, ranges, &range_end); end = std::max(end, range_end); }
- The MapColumnReader, It is very similar to ListColumnReader, Just keep Map<Struct<key,value>> in mind.
- up to Now, complex type as value type is not supported, as the same Array<Array
> is not supported either, I will support it in another pr.
Checklist:
- [x] I have added test cases for my bug fix or my new feature
- [ ] I have added user document for my new feature or new function