quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Published 20 hours ago • verified publisher icon quickwit-oss

Reorganize directories

Open fulmicoton opened this issue 2 years ago • 1 comments

We could organize by role

  • ingester
  • indexer
  • searcher

Durability nature

  • temporary data
  • caches
  • storages

fulmicoton avatar Sep 21 '23 00:09 fulmicoton

The ephemeral vs persistent dimension is interesting. For instance, you could mount a local SSD on qwdata/ephemeral to serialize splits and cache them for merges, but mount qw/persistent on an EBS to store the WAL.

Maybe this is what we want: qwdata/{ephemeral,persistent}/<role>/<purpose>:

qwdata/
    ephemeral/
        indexer/
        searcher/
    persistent/
        ingester/
            wal/
        metastore/
            indexes/
            manifest.json

guilload avatar Apr 18 '24 16:04 guilload