memtier_benchmark icon indicating copy to clipboard operation
memtier_benchmark copied to clipboard

improve the performance of imported_keylist deconstruction

Open DarrenJiang13 opened this issue 2 years ago • 2 comments

Problem

When I run memtier_benchmark with --data-import (including 500000 keys), it took a long time to quit after benchmark finished. With pstack, I found the process was waiting in ~imported_keylist()

...
#6  std::vector<imported_keylist::key*, std::allocator<imported_keylist::key*> >::erase (__position=..., this=0x1e685e8) at /usr/include/c++/9/bits/stl_vector.h:1428
#7  imported_keylist::~imported_keylist (this=0x1e685e0, __in_chrg=<optimized out>) at obj_gen.cpp:534
#8  0x0000000000407a1a in main (argc=<optimized out>, argv=<optimized out>) at memtier_benchmark.cpp:1641

Solution

erase() one by one for a dead vector wastes some time. So I replaced erase() with clear(), which makes it much faster.

DarrenJiang13 avatar Oct 24 '23 08:10 DarrenJiang13

Test command

./memtier_benchmark -p 6379 -t 2 -c 8 -n 500000 --ratio 1:0 --data-import ./dump100.csv

Test result:

key numbers time cost to quit after finishing benchmark (second)
before commit 500000 175
after commit 500000 1

Supplement

dump100.csv.zip

DarrenJiang13 avatar Nov 03 '23 07:11 DarrenJiang13

@filipecosta90 hi could you please check this PR? I think it is ready to be merged

DarrenJiang13 avatar Jan 18 '24 09:01 DarrenJiang13