quickwit-oss
Add min/max statistics to the metastore for pruning
Is your feature request related to a problem? Please describe. The current pruning mechanisms are time, tag and partitioning. We often run into queries that end up having to target every split in the index because we can leverage neither of these. A typical example is when we have two time dimensions, event time and ingestion time. Only one can be used as time in QW, but we might want to query on the second one.
Describe the solution you'd like
I would like to add a configuration (e.g stats_fields) in the doc mapping similar to tag_fields.
- it would be only compatible with numerical fields
- when packaging the split, we would compute min and max and add it to the metadata
- we would store the min and max in the metastore by either
- adding a field to split_metadata_json. This is simple to setup, we could probably still push down the pruning to the metastore but it would be quite expensive.
- using an encoding similar to tags, with something like a Vec<String> where min and max values would be encoded
- use a JSON type
Describe alternatives you've considered We can come up with workarounds using tags. For instance for the problem of the secondary time dimension, we could record the ingestion day as tag. But its less flexible, harder to setup for users, and more costly.
@fmassot @fulmicoton I would swear that I already saw an issue about this but I couldn't find it. What do you think about this feature? Would you be interested in it? We might chose a different way of encoding the min/max in the metastore depending on whether we can merge it upstream (if not, we'll stick with sticking it into split_metadata_json to keep the metastore schema in sync).
Can't we push that to tantivy? Sooner or later the ParadeDB folks will be after that feature too. On the Quickwit side, as long as the stats make it to the hotache, we should be good.
I guess we could.
A few limitations:
- For indexes with hundreds of thousands of splits, you really want to have as many metastore level pruning options as possible. After a searcher upgrade, the cache is emptied, and a query could generate hundreds of thousands of GET requests just to load the hotcaches (object store RPS is really a big challenge for us)
- AFAIK, we currently don't have a stage for partial query resolution against the hotcache, we would need to add that
- Does Tantivy really want to go in that direction (only Paul can answer)
I think we have narrowed down the need to perform this metastore pruning only on a secondary time dimension. As for most event streams, we have two timestamps for our events:
- ingest time
- event time
We use event time as timestamp because it's usually what our users are interested in. But there are a couple of more "operational" use cases that work based in ingest time (e.g show to users the amount of data they ingested, verify volumes of logs ingested).
We know one other large scale QW user that has a fork with the same change.
@guilload / @fmassot would you be interested for us to contribute this change? Otherwise we'll do it in a fork, but it will be a bit tedious to maintain.