RedisShake
RedisShake copied to clipboard
Published
20 hours ago •
tair-opensource
tair-opensource
Error when synchronizing Redis 4.0 standalone to redis 5.0 cluster
- [x] 请确保已经看过 wiki：https://github.com/alibaba/RedisShake/wiki
- [x] 请确保已经学习过 Markdown 语法，良好的排版有助于维护人员了解你的问题
- [x] 请在此提供足够的信息供社区维护人员排查问题
- [x] 请在提交 issue 前删除此模板中多余的文字，包括这几句话
sync.toml
type = "sync"
[source]
version = 4.0 # redis version, such as 2.8, 4.0, 5.0, 6.0, 6.2, 7.0, ...
address = "10.106.xx.xxx:6379"
username = "" # keep empty if not using ACL
password = "" # keep empty if no authentication is required
tls = false
elasticache_psync = "" # using when source is ElastiCache. ref: https://github.com/alibaba/RedisShake/issues/373
[target]
type = "cluster" # "standalone" or "cluster"
version = 5.0 # redis version, such as 2.8, 4.0, 5.0, 6.0, 6.2, 7.0, ...
# When the target is a cluster, write the address of one of the nodes.
# redis-shake will obtain other nodes through the `cluster nodes` command.
address = "10.174.xx.xxx:11309"
username = "" # keep empty if not using ACL
password = "" # keep empty if no authentication is required
tls = false
[advanced]
dir = "data"
# runtime.GOMAXPROCS, 0 means use runtime.NumCPU() cpu cores
ncpu = 4
# pprof port, 0 means disable
pprof_port = 0
# metric port, 0 means disable
metrics_port = 0
# log
log_file = "redis-shake.log"
log_level = "info" # debug, info or warn
log_interval = 5 # in seconds
# redis-shake gets key and value from rdb file, and uses RESTORE command to
# create the key in target redis. Redis RESTORE will return a "Target key name
# is busy" error when key already exists. You can use this configuration item
# to change the default behavior of restore:
# panic: redis-shake will stop when meet "Target key name is busy" error.
# rewrite: redis-shake will replace the key with new value.
# ignore: redis-shake will skip restore the key when meet "Target key name is busy" error.
rdb_restore_command_behavior = "rewrite" # panic, rewrite or skip
# pipeline
pipeline_count_limit = 1024
# Client query buffers accumulate new commands. They are limited to a fixed
# amount by default. This amount is normally 1gb.
target_redis_client_max_querybuf_len = 1024_000_000
# In the Redis protocol, bulk requests, that are, elements representing single
# strings, are normally limited to 512 mb.
target_redis_proto_max_bulk_len = 512_000_000
CLI
sudo ./bin/redis-shake sync.toml
LOG
2022-12-16 16:11:40 INF GOOS: darwin, GOARCH: arm64
2022-12-16 16:11:40 INF Ncpu: 4, GOMAXPROCS: 4
2022-12-16 16:11:40 INF pid: 84003
2022-12-16 16:11:40 INF pprof_port: 0
2022-12-16 16:11:40 INF No lua file specified, will not filter any cmd.
2022-12-16 16:11:40 INF no password. address=[10.174.xx.xxx:11309]
2022-12-16 16:11:40 INF redisClusterWriter load cluster nodes. line=10a174a54a214a98211000000000000000000024 10.174.xx.xxx:11289@11289 master - 1671174700348 1671174700348 99 connected 12288-16383
2022-12-16 16:11:40 INF no password. address=[10.174.xx.xxx:11289]
2022-12-16 16:11:40 INF redisWriter connected to redis successful. address=[10.174.xx.xxx:11289]
2022-12-16 16:11:40 INF redisClusterWriter load cluster nodes. line=10a174a54a209a90311000000000000000000000 10.174.xx.xxx:11309@11309 myself,master - 1671174700348 1671174700348 99 connected 0-4095
2022-12-16 16:11:40 INF no password. address=[10.174.xx.xxx:11309]
2022-12-16 16:11:40 INF redisWriter connected to redis successful. address=[10.174.xx.xxx:11309]
2022-12-16 16:11:40 INF redisClusterWriter load cluster nodes. line=10a174a54a215a98211000000000000000000008 10.174.xx.xxx:11289@11289 master - 1671174700348 1671174700348 99 connected 4096-8191
2022-12-16 16:11:40 INF no password. address=[10.174.xx.xxx:11289]
2022-12-16 16:11:40 INF redisWriter connected to redis successful. address=[10.174.xx.xxx:11289]
2022-12-16 16:11:40 INF redisClusterWriter load cluster nodes. line=10a174a54a216a93211000000000000000000016 10.174.xx.xxx:11239@11239 master - 1671174700348 1671174700348 99 connected 8192-12287
2022-12-16 16:11:40 INF no password. address=[10.174.xx.xxx:11239]
2022-12-16 16:11:40 INF redisWriter connected to redis successful. address=[10.174.xx.xxx:11239]
2022-12-16 16:11:40 INF redisClusterWriter connected to redis cluster successful. addresses=[10.174.xx.xxx:11289 10.174.xx.xxx:11309 10.174.xx.xxx:11289 10.174.xx.xxx:11239]
2022-12-16 16:11:40 INF no password. address=[10.106.xx.xxx:6379]
2022-12-16 16:11:40 INF psyncReader connected to redis successful. address=[10.106.xx.xxx:6379]
2022-12-16 16:11:40 WRN remove file. filename=[220782554095.aof]
2022-12-16 16:11:40 WRN remove file. filename=[dump.rdb]
2022-12-16 16:11:40 INF start save RDB. address=[10.106.xx.xxx:6379]
2022-12-16 16:11:40 INF send [replconf listening-port 10007]
2022-12-16 16:11:40 INF send [PSYNC ? -1]
2022-12-16 16:11:40 INF receive [FULLRESYNC ce6bf38546f40d30f619158710725fc42c1496de 220792215831]
2022-12-16 16:11:40 INF source db is doing bgsave. address=[10.106.xx.xxx:6379]
2022-12-16 16:11:40 INF source db bgsave finished. timeUsed=[0.54]s, address=[10.106.xx.xxx:6379]
2022-12-16 16:11:40 INF received rdb length. length=[18161262]
2022-12-16 16:11:40 INF create dump.rdb file. filename_path=[dump.rdb]
2022-12-16 16:11:43 INF save RDB finished. address=[10.106.xx.xxx:6379], total_bytes=[18161262]
2022-12-16 16:11:43 INF start send RDB. address=[10.106.xx.xxx:6379]
2022-12-16 16:11:43 INF start save AOF. address=[10.106.xx.xxx:6379]
2022-12-16 16:11:43 INF RDB version: 8
2022-12-16 16:11:43 INF RDB AUX fields. key=[redis-ver], value=[4.0.8]
2022-12-16 16:11:43 INF RDB AUX fields. key=[redis-bits], value=[64]
2022-12-16 16:11:43 INF RDB AUX fields. key=[ctime], value=[1671174700]
2022-12-16 16:11:43 INF RDB AUX fields. key=[used-mem], value=[65444160]
2022-12-16 16:11:43 INF AOFWriter open file. filename=[220792215831.aof]
2022-12-16 16:11:43 INF RDB repl-stream-db: 0
2022-12-16 16:11:43 INF RDB AUX fields. key=[repl-id], value=[ce6bf38546f40d30f619158710725fc42c1496de]
2022-12-16 16:11:43 INF RDB AUX fields. key=[repl-offset], value=[220792215831]
2022-12-16 16:11:43 INF RDB AUX fields. key=[aof-preamble], value=[0]
2022-12-16 16:11:43 INF RDB resize db. db_size=[38604], expire_size=[371]
2022-12-16 16:11:45 INF syncing rdb. percent=[42.11]%, allowOps=[4408.40], disallowOps=[0.00], entryId=[22042], InQueueEntriesCount=[16], unansweredBytesCount=[0]bytes, rdbFileSize=[0.017]G, rdbSendSize=[0.007]G
2022-12-16 16:11:46 INF send RDB finished. address=[10.106.xx.xxx:6379], repl-stream-db=[0]
2022-12-16 16:11:47 INF AOFReader open file. aof_filename=[220792215831.aof]
2022-12-16 16:11:47 PNC redisWriter received error. error=[ERR Unsupported command], argv=[PUBLISH __sentinel__:hello 10.106.xx.xxx,26379,8c54fa8b3b2bde3acaaffbdc94d87e650d271198,33,mymaster,10.106.xx.xxx,6379,19], slots=[], reply=[<nil>]
panic: redisWriter received error. error=[ERR Unsupported command], argv=[PUBLISH __sentinel__:hello 10.106.xx.xxx,26379,8c54fa8b3b2bde3acaaffbdc94d87e650d271198,33,mymaster,10.106.xx.xxx,6379,19], slots=[], reply=[<nil>]
goroutine 20 [running]:
github.com/rs/zerolog.(*Logger).Panic.func1({0x1400023ab60?, 0x0?})
github.com/rs/[email protected]/log.go:359 +0x30
github.com/rs/zerolog.(*Event).msg(0x14000090660, {0x1400023ab60, 0xd6})
github.com/rs/[email protected]/event.go:156 +0x244
github.com/rs/zerolog.(*Event).Msg(...)
github.com/rs/[email protected]/event.go:108
github.com/alibaba/RedisShake/internal/log.logFinally(0x14000090660, {0x104e5e835?, 0x18?}, {0x1400025ff68?, 0x1400004ee01?, 0x104e0613c?})
github.com/alibaba/RedisShake/internal/log/func.go:77 +0x60
github.com/alibaba/RedisShake/internal/log.Panicf({0x104e5e835, 0x45}, {0x1400025ff68, 0x4, 0x4})
github.com/alibaba/RedisShake/internal/log/func.go:27 +0x50
github.com/alibaba/RedisShake/internal/writer.(*redisWriter).flushInterval(0x1400007c140)
github.com/alibaba/RedisShake/internal/writer/redis.go:75 +0x2f8
created by github.com/alibaba/RedisShake/internal/writer.NewRedisWriter
github.com/alibaba/RedisShake/internal/writer/redis.go:35 +0x1a0
Although the source and target are different, I have synced in the same environment before.
This time the error pops up like this. Can you tell me what this error log means?
The difference from that time is that this cluster has all different IPs.
The meaning of the error is: the target does not support the "PUBLISH" command.
error=[ERR Unsupported command], argv=[PUBLISH sentinel:hello 1***]
Is the target a standard Redis cluster, and whether there are components such as proxy?
@suxb201
I found out that pub/sub is not supported due to the difference in cluster topology.
Is there any way to sync in this situation?
@junhok82 Use filter to filter all PUBSUB commands:
-- skip all pub-sub commands
function filter(id, is_base, group, cmd_name, keys, slots, db_id, timestamp_ms)
if group == "PUBSUB" then
return 1, db_id -- disallow
else
return 0, db_id -- allow
end
end
More docs: https://github.com/alibaba/RedisShake/wiki/%E4%BD%BF%E7%94%A8-filters-%E5%81%9A%E6%95%B0%E6%8D%AE%E6%B8%85%E6%B4%97