quickwit
quickwit copied to clipboard
Remove RocksDB dependency
A Quickwit node is meant to support thousands of indexes, while maintaining a decent throughput, and small memory footprint.
Currently the Ingest API works by using RocksDB as our storage. Each queue gets its own column family.
When we publish a split, we truncate it. RocksDB allows efficient truncation. If a SSTable is entirely deleted, the truncation will just delete the file instead of using tombstone & compaction. In my experience it is however unable to truncate the WAL, so the current solution ends up creation 1000+ files.
Also the rocksdb dependency is making building portable binaries very difficult, and the build is much slower because of it.
The following project aims at replacing the current RocksDB-based solution. https://github.com/quickwit-oss/mrecordlog/tree/main/src
#1898 shows what a naive replacement could look like.