KeyDB
KeyDB copied to clipboard
Snapchat
When updating FLASH values use bulk insert
Bulk insert is dramatically more performant than multiple single inserts.
Perf benchmarks TBD
@JohnSully @msotheeswaran John and Malavan, i picked up these 2 commits on top of the latest master branch and compile the code in flash as the following: make ENABLE_FLASH=yes
and then i ran the ./runtest unit tests, i am seeing the following replication tcl unit test failure. please help to look at this issue. Thanks.
[ok]: First server should have role slave after SLAVEOF [ok]: Big Hash table: SORT BY key with limit [ok]: MIGRATE will not overwrite existing keys, unless REPLACE is used I/O error reading reply while executing "{}$r sadd $k $v" ("uplevel" body line 2) invoked from within "uplevel 1 [lindex $args $path]" (procedure "randpath" line 3) invoked from within "randpath { {}$r set $k $v } { {}$r lpush $k $v } { {}$r sadd $k $v ..." (procedure "createComplexDataset" line 30) invoked from within "createComplexDataset $r $ops" (procedure "bg_complex_data" line 5) invoked from within "bg_complex_data [lindex $argv 0] [lindex $argv 1] [lindex $argv 2] [lindex $argv 3] [lindex $argv 4]" (file "tests/helpers/bg_complex_data.tcl" line 13) I/O error reading reply while executing "{*}$r type $k" (procedure "createComplexDataset" line 27) invoked from within "createComplexDataset $r $ops" (procedure "bg_complex_data" line 5) invoked from within "bg_complex_data [lindex $argv 0] [lindex $argv 1] [lindex $argv 2] [lindex $argv 3] [lindex $argv 4]" (file "tests/helpers/bg_complex_data.tcl" line 13) I/O error reading reply while executing "$r client setname LOAD_HANDLER" (procedure "bg_complex_data" line 3) invoked from within "bg_complex_data [lindex $argv 0] [lindex $argv 1] [lindex $argv 2] [lindex $argv 3] [lindex $argv 4]" (file "tests/helpers/bg_complex_data.tcl" line 13) [ok]: MIGRATE propagates TTL correctly [ok]: EXPIREMEMBER works (zset) [ok]: TTL for subkey expires works
Hey Paul,
I pushed a fix to the crashing test.
@JohnSully @msotheeswaran Hello John, thanks for the change, runtest no longer crashes, however, I am seeing a few unit test failures and "Slave not correctly synchronized" in replication-psync.tcl. The errors are listed below:
[err]: Test replication partial resync: no backlog (diskless: no, disabled, reconnect: 1) in tests/integration/replication-psync-flash.tcl Slave not correctly synchronized [ok]: Slave should be able to synchronize with the master [ok]: Detect write load to master [err]: Test replication partial resync: ok after delay (diskless: no, disabled, reconnect: 1) in tests/integration/replication-psync-flash.tcl Slave not correctly synchronized [ok]: Slave should be able to synchronize with the master [ok]: Detect write load to master [err]: Test replication partial resync: backlog expired (diskless: no, disabled, reconnect: 1) in tests/integration/replication-psync-flash.tcl Slave not correctly synchronized [ok]: Slave should be able to synchronize with the master [ok]: Detect write load to master [ok]: Test replication partial resync: no reconnection, just sync (diskless: no, swapdb, reconnect: 0) [ok]: Slave should be able to synchronize with the master [ok]: Detect write load to master
@msotheeswaran Malavan, The following error appears when running runtest unit tests against the flash on the main branch as well, so it appears that the following unit test failure is caused by the previous MR (Not by this commit). It would be helpful if you could investigate it. I appreciate your help.
-- ran runtest using the latest code from Main after compiling the code in flash --- [err]: Test replication partial resync: no backlog (diskless: no, disabled, reconnect: 1) in tests/integration/replication-psync-flash.tcl Slave not correctly synchronized
@paulmchen #604 should fix the tests
@msotheeswaran verified and all are passed. Thanks.
Releasing this will make the flash usable. Right now, its not usable because of the high IO writes on EBS.
