LocustDB
LocustDB copied to clipboard
cswinter
Blazingly fast analytics database that will rapidly devour all of your data.
Columns are reordered to have projections followed by aggregations.
There are a lot of cases where intermediary results could be streamed between operators, but aren't. One example is inability to stream from operators that produce a full output. There...
There is one remaining place during query execution which is performed "manually" rather than by constructing and executing a query plan: https://github.com/cswinter/LocustDB/blob/38ba86d6f915e1173d9075f2a0fc206d674441c6/src/engine/execution/batch_merging.rs#L247 Mostly this just requires moving the `Data::append_all` function...
Various parts of the code use `mem::transmute` to make lifetimes work out for data referencing strings. More details on the issue here: https://clemenswinter.com/2018/07/09/how-to-analyze-billions-of-records-per-second-on-a-single-desktop-pc/4/ Probably not worth revisiting this until generic...
The tablename should be part of the RocksDB primary key so scans over a column that exists in multiple tables doesn't read unnecessary blocks.
Performance might benefit from ReadOptions.readahead and ReadOptions.iterate_upper_bound once those are exposed by rust-rocksdb
Columns that contain both strings, ints and null values. - [ ] when loading from CSV - [ ] in select clause - [ ] in group by clause -...
Note this portion of the query plan which combines passenger_count with itself: ``` casted_1 = column_0 as I64 TypeConversionOperator bitpacked_2 = casted_1 + (casted_1 :explain SELECT passenger_count, count(0) FROM trips...