quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

allow updating of doc mapper

Open trinity-1686a opened this issue 1 year ago • 3 comments

Description

this pr aims at laying the foundation work for updating the doc mapping of an index

  • [x] store a doc mapper version in the doc mapper
  • [x] store a doc mapper version in splits meta
  • [ ] allow updating the doc mapper, auto-incrementing its version (metastore level)
  • [ ] allow updating the doc mapper (api level)
  • [x] block merges between multiple version of a doc mapper
  • [ ] restart ingestion pipeline so they notice the change in doc mapper decently quickly (stretch goal, maybe put in a 2nd pull request)

How was this PR tested?

  • [x] test we properly block merges (unit tests)
  • [ ] test we properly block merges (integration tests)
  • [x] test modifying different properties (tokenizer, what is stored...), and how it breaks (or not) search

trinity-1686a avatar Apr 30 '24 15:04 trinity-1686a

We should probably merge https://github.com/quickwit-oss/quickwit/pull/4868 before addressing:

  • allow updating the doc mapper (api level)

rdettai avatar May 06 '24 09:05 rdettai

we are going to ship this in multiple smaller PR, which means this should be ready for review

trinity-1686a avatar Jun 05 '24 14:06 trinity-1686a

On SSD:

Average search latency is 0.99x that of the reference (lower is better).
Ref run id: 1883, ref commit: 01571db8d48f2ce474aa96cee91ded62938b0805
Link

On GCS:

Average search latency is 1.0x that of the reference (lower is better).
Ref run id: 1884, ref commit: 01571db8d48f2ce474aa96cee91ded62938b0805
Link

github-actions[bot] avatar Jun 05 '24 14:06 github-actions[bot]

made hard to work on by addition of doc_mapping_uid in a different PR. Remade from scratch in #5218

trinity-1686a avatar Jul 12 '24 17:07 trinity-1686a