[NEW] Support Redis 7 (RDB format version 10) dumps

[Niek]

I get this error when trying to load a dump file generated by Redis 7:

26966:26966:M 27 Dec 2022 05:27:30.791 # Can't handle RDB format version 10
26966:26966:M 27 Dec 2022 05:27:30.791 # Fatal error loading the DB: Invalid argument. Exiting.

Is support for this planned?

[air3ijai]

Got the same error on local RDB load

1:1:S 27 Dec 2022 13:11:02.432 # Server initialized
1:1:S 27 Dec 2022 13:11:02.432 * <bf> RedisBloom version 2.2.18 (Git=8b6ee3b)
1:1:S 27 Dec 2022 13:11:02.432 * Module 'bf' loaded from /usr/lib/redis/modules/redisbloom.so
1:1:S 27 Dec 2022 13:11:02.433 # Can't handle RDB format version 10
1:1:S 27 Dec 2022 13:11:02.433 # Fatal error loading the DB: Invalid argument. Exiting.

And remote replication

1:16:S 27 Dec 2022 14:07:17.626 * REPLICAOF xx.xx.xx.xx:6379 enabled (user request from 'id=953 addr=127.0.0.1:34626 laddr=127.0.0.1:6379 fd=22 name= age=21 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=26 obl=0 oll=0 omem=0 tot-mem=61490 events=r cmd=replicaof user=default redir=-1')
1:16:S 27 Dec 2022 14:07:17.939 * Connecting to MASTER xx.xx.xx.xx:6379
1:16:S 27 Dec 2022 14:07:17.940 * MASTER <-> REPLICA sync started
1:16:S 27 Dec 2022 14:07:18.945 * Non blocking connect for SYNC fired the event.
1:16:S 27 Dec 2022 14:07:18.947 * Master does not support REPLPING, sending PING instead...
1:16:S 27 Dec 2022 14:07:18.947 * Non blocking connect for SYNC fired the event.
1:16:S 27 Dec 2022 14:07:18.947 * Master replied to PING, replication can continue...
1:16:S 27 Dec 2022 14:07:18.948 # non-fatal: Master doesn't understand REPLCONF uuid
1:16:S 27 Dec 2022 14:07:18.948 * Partial resynchronization not possible (no cached master)
1:16:S 27 Dec 2022 14:07:43.433 * Full resync from master: 5f9e65a03498f1a54fe3b078e2dc78e41735f3e8:1316
1:16:S 27 Dec 2022 14:07:44.338 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:16:S 27 Dec 2022 14:11:57.266 * MASTER <-> REPLICA sync: Loading DB in memory
1:16:S 27 Dec 2022 14:11:57.278 # Can't handle RDB format version 10
1:16:S 27 Dec 2022 14:11:57.278 # Failed trying to load the MASTER synchronization DB from disk

[doptime]

I finally solved same problem by writting a new migrate python script: https://github.com/yangkequn/RedisTools/blob/master/redisMigrateToKeydb.py

[air3ijai]

[QUESTION] How to migrate RDB to new format supported by Redis 7? #11172

[air3ijai]

Redis v7 to KeyDB v6 migration using redis-rdb-cli

1. Install redis-rdb-cli on KeyDB 6 node

   # Prerequisites
   sudo apt update
   sudo apt install wget unzip openjdk-8-jre-headless
   
   # Redis rdb CLI
   cd /opt
   wget https://github.com/leonchen83/redis-rdb-cli/releases/latest/download/redis-rdb-cli-release.zip
   unzip redis-rdb-cli-release.zip
   

2. Update redis-rdb-cli config to downgrade the version during migration

   vi /opt/redis-rdb-cli/conf/redis-rdb-cli.conf
   
   dump_rdb_version=9
   

3. Check for the big keys

   # Create the report with 10 biggest keys
   /opt/redis-rdb-cli/bin/rct -f mem -s redis://redis-7:6379?authPassword=password -o /data/dump.mem -l 10
   
   # Check the report
   head /data/dump.mem
   database,type,key,size_in_bytes,encoding,num_elements,len_largest_element,expiry
   0,module,"tree:20:1","792.1MB",module2,1,"792.1MB",""
   0,module,"tree:22:1","792.1MB",module2,1,"792.1MB",""
   0,module,"tree:21:1","792.1MB",module2,1,"792.1MB",""
   0,module,"tree:17:1","484.9MB",module2,1,"484.9MB",""
   0,module,"tree:19:1","484.9MB",module2,1,"484.9MB",""
   0,module,"tree:18:1","484.9MB",module2,1,"484.9MB",""
   0,module,"tree:15:1","484.9MB",module2,1,"484.9MB",""
   0,module,"tree:16:1","484.9MB",module2,1,"484.9MB",""
   0,module,"tree:14:1","480.1MB",module2,1,"480.1MB",""
   
   # If you have keys bigger than 512MB
   # by default redis-rdb-cli use 4 threads to migrate data to target.
   # Xms, Xmx at least 4*{max key size}
   # 4 * 792.1 = 3168.4m
   export JAVA_TOOL_OPTIONS="-Xms4g -Xmx4g"
   
   # Permit big keys during import on KeyDB 6
   keydb-cli -h keydb-6 -p 6379 -a password \
     config set proto-max-bulk-len 1024mb
   

4. Disable dumps during import on KeyDB 6

   keydb-cli -h keydb-6 -p 6379 -a password \
     config set save ""
   

5. Migrate the data from Redis 7 to KeyDB 6

   /opt/redis-rdb-cli/bin/rmt \
     -s redis://redis-7:6379?authPassword=password \
     -m redis://keydb-6:6379?authPassword=password
   

   /[ 20.4GB| 80.1MB/s]
   

   real	4m15.967s
   

6. Compare the datasets

   redis-cli -h redis-7 -p 6379 -a password \
     info keyspace
   
   keydb-cli -h keydb-6 -p 6379 -a password \
     info keyspace
   

   # Redis 7
   # Keyspace
   db0:keys=1994658,expires=0,avg_ttl=0
   
   # KeyDB 6
   # Keyspace
   db0:keys=1994658,expires=0,avg_ttl=0
   

7. Change back KeyDB settings and perform a backup

   keydb-cli -h keydb-6 -p 6379 -a password \
     config set proto-max-bulk-len 512mb
   
   keydb-cli -h keydb-6 -p 6379 -a password \
     save 3600 1000
   
   keydb-cli -h keydb-6 -p 6379 -a password \
     bgsave
   

Based on the keys number we can state that all the data was successfully migrated.

Please also see Check migrated data consistency #40.

More information about AOF migration you can find in the Sync data between KeyDB v6 - Redis v7 #38

[TomHellier]

#420 - existing ticket to update keydb to v7