quickwit
quickwit copied to clipboard
allow updating of doc mapper
Description
this pr aims at laying the foundation work for updating the doc mapping of an index
- [x] store a doc mapper version in the doc mapper
- [x] store a doc mapper version in splits meta
- [ ] allow updating the doc mapper, auto-incrementing its version (metastore level)
- [ ] allow updating the doc mapper (api level)
- [x] block merges between multiple version of a doc mapper
- [ ] restart ingestion pipeline so they notice the change in doc mapper decently quickly (stretch goal, maybe put in a 2nd pull request)
How was this PR tested?
- [x] test we properly block merges (unit tests)
- [ ] test we properly block merges (integration tests)
- [x] test modifying different properties (tokenizer, what is stored...), and how it breaks (or not) search
We should probably merge https://github.com/quickwit-oss/quickwit/pull/4868 before addressing:
allow updating the doc mapper (api level)
we are going to ship this in multiple smaller PR, which means this should be ready for review
On SSD:
Average search latency is 0.99x that of the reference (lower is better).
Ref run id: 1883, ref commit: 01571db8d48f2ce474aa96cee91ded62938b0805
Link
On GCS:
Average search latency is 1.0x that of the reference (lower is better).
Ref run id: 1884, ref commit: 01571db8d48f2ce474aa96cee91ded62938b0805
Link
made hard to work on by addition of doc_mapping_uid in a different PR. Remade from scratch in #5218