[WIP] Support map type

[xiangfu0]

Based on the branch: https://github.com/ege-st/pinot/tree/erich_feature_map_type_merge_mutable_immutable_-_squash_v2_merge_master_squash

[siddharthteotia]

Is this about supporting complex type MAP as a first class data type in Pinot ?

I am interested in helping review this. Is there a rough design doc / high level idea that can be shared to understand the semantics of how this will look in storage and query wise ?

Are we planning to store this as a raw blob / binary bytes in forward index ?

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 64.88920% with 507 lines in your changes missing coverage. Please review.

Project coverage is 62.00%. Comparing base (59551e4) to head (0229ca1). Report is 861 commits behind head on master.

Files	Patch %	Lines
...t/local/realtime/impl/map/MutableMapIndexImpl.java	58.01%	76 Missing and 13 partials :warning:
...ocal/segment/creator/impl/map/MapIndexCreator.java	79.63%	45 Missing and 11 partials :warning:
...egment/local/segment/index/map/NullDataSource.java	0.00%	41 Missing :warning:
...or/impl/stats/MapColumnPreIndexStatsCollector.java	63.10%	29 Missing and 9 partials :warning:
.../apache/pinot/spi/config/table/MapIndexConfig.java	0.00%	38 Missing :warning:
...r/transform/function/MapItemTransformFunction.java	0.00%	37 Missing :warning:
...ent/index/readers/map/ImmutableMapIndexReader.java	48.61%	34 Missing and 3 partials :warning:
...ocal/segment/index/map/ImmutableMapDataSource.java	37.83%	22 Missing and 1 partial :warning:
...local/segment/creator/impl/map/DenseMapHeader.java	89.14%	5 Missing and 14 partials :warning:
...local/segment/creator/impl/map/MapIndexHeader.java	85.83%	13 Missing and 4 partials :warning:
... and 26 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #13453      +/-   ##
============================================
+ Coverage     61.75%   62.00%   +0.24%     
+ Complexity      207      198       -9     
============================================
  Files          2436     2574     +138     
  Lines        133233   142514    +9281     
  Branches      20636    22104    +1468     
============================================
+ Hits          82274    88360    +6086     
- Misses        44911    47435    +2524     
- Partials       6048     6719     +671     

Flag	Coverage Δ
custom-integration1	<0.01% <0.00%> (-0.01%)	:arrow_down:
integration	<0.01% <0.00%> (-0.01%)	:arrow_down:
integration1	<0.01% <0.00%> (-0.01%)	:arrow_down:
integration2	0.00% <0.00%> (ø)
java-11	61.94% <64.88%> (+0.24%)	:arrow_up:
java-21	61.89% <64.88%> (+0.26%)	:arrow_up:
skip-bytebuffers-false	61.98% <64.88%> (+0.23%)	:arrow_up:
skip-bytebuffers-true	61.85% <64.88%> (+34.12%)	:arrow_up:
temurin	62.00% <64.88%> (+0.24%)	:arrow_up:
unittests	61.99% <64.88%> (+0.24%)	:arrow_up:
unittests1	45.88% <7.20%> (-1.01%)	:arrow_down:
unittests2	28.10% <57.68%> (+0.37%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[xiangfu0]

Is this about supporting complex type MAP as a first class data type in Pinot ?

I am interested in helping review this. Is there a rough design doc / high level idea that can be shared to understand the semantics of how this will look in storage and query wise ?

Are we planning to store this as a raw blob / binary bytes in forward index ?

Thanks Sid! Wilk share a design doc soon. The basic idea here is to store map of key to any primitive types using forward index.

And later on we can add more index support for map item function based on the density etc

[siddharthteotia]

Sounds good. Thanks @xiangfu0 . Please share a design doc when you can.

Long back, I had done a prototype of MAP and LIST support, essentially composing these where we can compose and store primitives and complex types in a flexible way. So, I am interested.