[Feature] Skipping Index in the New Columned-Based Storage

[hanahmily]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

This issue proposes the introduction of a skipping index in the new columned-based storage for Measure and Stream. The skipping index is intended to help filter the data more efficiently. The index will support two types: Min-Max and Bloom filter.

The proposed index will support two types:

1. Min-Max Index: This index will store the minimum and maximum values of each column in each data block. It will quickly eliminate the data blocks that do not contain the desired range of values. All numeric data will applied to this type.

2. Bloom Filter Index: This probabilistic data structure will help in determining whether an element may be in a set or not. It will reduce the I/O operations by avoiding unnecessary disk reads for data that are not present. All data will applied to this type.

The numeric data includes both types at the same time.

In the API, we propose to add a new index rule type: "skipping".

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• [ ] Yes I am willing to submit a pull request on my own!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[zesiar0]

please assign to me