quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Remove RocksDB dependency

Open fulmicoton opened this issue 3 years ago • 0 comments

A Quickwit node is meant to support thousands of indexes, while maintaining a decent throughput, and small memory footprint.

Currently the Ingest API works by using RocksDB as our storage. Each queue gets its own column family.

When we publish a split, we truncate it. RocksDB allows efficient truncation. If a SSTable is entirely deleted, the truncation will just delete the file instead of using tombstone & compaction. In my experience it is however unable to truncate the WAL, so the current solution ends up creation 1000+ files.

Also the rocksdb dependency is making building portable binaries very difficult, and the build is much slower because of it.

The following project aims at replacing the current RocksDB-based solution. https://github.com/quickwit-oss/mrecordlog/tree/main/src

#1898 shows what a naive replacement could look like.

fulmicoton avatar Oct 06 '22 06:10 fulmicoton