ArcticDB
ArcticDB copied to clipboard
Use mimalloc everywhere
Profiling our performance on the billion row challenge showed deallocating segments to be the major bottleneck. Code to generate the data and profile the grouping available on this branch:
https://github.com/man-group/ArcticDB/tree/brc-speedup
Run from the python
directory as:
python -m pytest -s tests/unit/arcticdb/version_store/profile_billion_row_challenge.py::test_groupby
Building the mimalloc library from the master
branch and then rerunning as:
LD_PRELOAD=../../mimalloc/out/release/libmimalloc.so python -m pytest -s tests/unit/arcticdb/version_store/profile_billion_row_challenge.py::test_groupby
resulted in a x20 speedup using 32 CPU threads and 48 IO threads.
Task is to use this as our allocator everywhere.