ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

Use mimalloc everywhere

Open alexowens90 opened this issue 9 months ago • 0 comments

Profiling our performance on the billion row challenge showed deallocating segments to be the major bottleneck. Code to generate the data and profile the grouping available on this branch: https://github.com/man-group/ArcticDB/tree/brc-speedup Run from the python directory as: python -m pytest -s tests/unit/arcticdb/version_store/profile_billion_row_challenge.py::test_groupby Building the mimalloc library from the master branch and then rerunning as: LD_PRELOAD=../../mimalloc/out/release/libmimalloc.so python -m pytest -s tests/unit/arcticdb/version_store/profile_billion_row_challenge.py::test_groupby resulted in a x20 speedup using 32 CPU threads and 48 IO threads. Task is to use this as our allocator everywhere.

alexowens90 avatar May 20 '24 14:05 alexowens90