starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Feature] support read map column from parquet file

Open zombee0 opened this issue 2 years ago • 0 comments

support read map column from parquet file

Signed-off-by: zombee0 [email protected]

What type of PR is this:

  • [ ] BugFix
  • [x] Feature
  • [ ] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Which issues of this PR fixes :

Fixes #10677

Problem Summary(Required) :

there are two main change:

  1. generate the IORange for map type column, the map type in parquet is represented by MAP<Struct<Key,Value>>, so we should get IORange by for (auto& field : schema_node->children[0].children) { int64_t range_end = 0; _collect_field_io_range(field, ranges, &range_end); end = std::max(end, range_end); }
  2. The MapColumnReader, It is very similar to ListColumnReader, Just keep Map<Struct<key,value>> in mind.
  3. up to Now, complex type as value type is not supported, as the same Array<Array> is not supported either, I will support it in another pr.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [ ] I have added user document for my new feature or new function

zombee0 avatar Sep 20 '22 02:09 zombee0