Bouncner
Bouncner
Please ignore the measurement above. Having problems to reproduce.
I think I did something major wrong with the numa pinning, but I am not sure. However, this is the current status on rapa: I cannot judge if the little...
Well, there are different things to discuss here. First a question: the 12.5 GB for SF 5, is this the max RSS you have seen? We do have a rather...
Oh, btw: it's not yet in the master branch, but we also support FSST (https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf) encoding. It can be directly merged if you're interested in it.
``` (release)> select sum(size_in_bytes) from meta_segments_accurate; === Columns |SUM(size_in_bytes)| | long| | null| === Chunk 0 === | | | 5107316678| ``` That's the data for SF 5. For TPC-H...
Yeah, I just verified it and it pretty much looks the same on my machine. After all the data loading is done, the console has around 6.5 GB of data...
In case you want to check what exactly is going on, colleagues have been successfully using Heaptrack for similar tasks: https://milianw.de/blog/heaptrack-a-heap-memory-profiler-for-linux.html
Yes, the memory footprint is definitely of high importance. 1 TB of raw data can easily be stored on 512 GB (assuming TPC-H data), but not with the default settings...
There are branches that tier segments to secondary storage (e.g., SSD) but they are not in a state where we recommend using them. Maybe might look different in about 6-8...