apache
[Feature] Introduce secondary index for paimon
Search before asking
- [X] I searched in the issues and found nothing similar.
Motivation
Up to now, Paimon use zorder & order sort compaction to speed up query. After sort compaction, files will be sorted by the order of specified columns. But in some situations, for example, we have tens of columns that should be added in the filter column, sometimes all of them come up together, sometimes, just a few of them. Zorder or order compaction can't handle this situation, because too many columns will reduce the effect of sorting. So if the column base number of these columns is small, we can use bloomfilter or other indexes to speed up queries. That's why this PIP comes up. I want to introduce a index framework to support paimon with flexible index system.
Solution
No response
Anything else?
No response
Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
The index should at the rowgroup(parquet) or stripe(orc) level for better
or
it can be configured at the file or row group level ？
