RedisShake
RedisShake copied to clipboard
Published
20 hours ago •
tair-opensource
tair-opensource
Redis Cluster同步数据到Codis3.2+Pika集群
问题描述
您好，我想咨询一下Redis同步数据到Pika的问题。
环境描述 源端 Redis 版本：4.0.14
目的端 Redis 版本：Codis 3.2+Pika 3.3.6 版本说明：https://github.com/kernelai/codis/releases
问题
看到Wiki中介绍，redis-shake可以支持Codis的数据迁移，如下：
想咨询下，具体target端的 type应该怎么配置呢？
@suxb201 非常感谢您的回复。 1、和您确认下是使用standalone方式配置单个codis-proxy的地址么？ 2、另外还有个问题，之前有看到codis不支持restore命令，只能在codis方面放开命令的白名单是吧？
- 我感觉可以，你可以试试
- 能开启的话开启最好，不能开启的话参考：https://github.com/tair-opensource/RedisShake/pull/594#issuecomment-1521082673
@suxb201 您好，我还想咨询下，我这边源端是75节点的redis cluster集群，目标端是codis，同步工具部署机器是一台物理机，配置是56c 256g内存，如果我使用推荐的 python3 cluster_helper.py ../redis-shake ../sync.toml 命令启动进程，会开启56个线程并行去同步数据吗？ 如果可以开启并发，可不可以使用参数限制，比如几个并发一起工作这种。 因为源端的redis 部署方式是单个物理机多实例，如果一起bgsave，恐怕要把机器磁盘直接拉死。 所以想咨询一下这个问题。 感谢。
2023-04-27 18:31:36 INF Ncpu: 8, GOMAXPROCS: 8
2023-04-27 18:31:36 INF pid: 1862927
2023-04-27 18:31:36 INF pprof_port: 0
2023-04-27 18:31:36 INF No lua file specified, will not filter any cmd.
2023-04-27 18:31:36 INF auth successful. address=[10.196.25.33:19000]
2023-04-27 18:31:36 INF redisWriter connected to redis successful. address=[10.196.25.33:19000]
2023-04-27 18:31:36 INF no password. address=[10.189.113.224:6379]
2023-04-27 18:31:36 INF psyncReader connected to redis successful. address=[10.189.113.224:6379]
2023-04-27 18:31:36 WRN remove file. filename=[50598975.aof]
2023-04-27 18:31:36 WRN remove file. filename=[dump.rdb]
2023-04-27 18:31:36 INF start save RDB. address=[10.189.113.224:6379]
2023-04-27 18:31:36 INF send [replconf listening-port 10007]
2023-04-27 18:31:36 INF send [PSYNC ? -1]
2023-04-27 18:31:37 INF receive [FULLRESYNC 3885ce44a970b69a023e8988a1ddecc22457a99b 1483008617703]
2023-04-27 18:31:37 INF source db is doing bgsave. address=[10.189.113.224:6379]
2023-04-27 18:31:41 INF source db is doing bgsave
2023-04-27 18:31:46 INF source db is doing bgsave
2023-04-27 18:31:51 INF source db is doing bgsave
2023-04-27 18:31:56 INF source db is doing bgsave
2023-04-27 18:32:01 INF source db is doing bgsave
2023-04-27 18:32:06 INF source db is doing bgsave
2023-04-27 18:32:11 INF source db is doing bgsave
2023-04-27 18:32:16 INF source db is doing bgsave
2023-04-27 18:32:21 INF source db is doing bgsave
2023-04-27 18:32:26 INF source db is doing bgsave
2023-04-27 18:32:31 INF source db is doing bgsave
2023-04-27 18:32:36 INF source db is doing bgsave
2023-04-27 18:32:41 INF source db is doing bgsave
2023-04-27 18:32:46 INF source db is doing bgsave
2023-04-27 18:32:51 INF source db is doing bgsave
2023-04-27 18:32:56 INF source db is doing bgsave
2023-04-27 18:33:01 INF source db is doing bgsave
2023-04-27 18:33:06 INF source db is doing bgsave
2023-04-27 18:33:11 INF source db is doing bgsave
2023-04-27 18:33:16 INF source db is doing bgsave
2023-04-27 18:33:21 INF source db is doing bgsave
2023-04-27 18:33:26 INF source db is doing bgsave
2023-04-27 18:33:31 INF source db is doing bgsave
2023-04-27 18:33:36 INF source db is doing bgsave
2023-04-27 18:33:41 INF source db is doing bgsave
2023-04-27 18:33:45 INF source db bgsave finished. timeUsed=[127.91]s, address=[10.189.113.224:6379]
2023-04-27 18:33:45 INF received rdb length. length=[7381713350]
2023-04-27 18:33:45 INF create dump.rdb file. filename_path=[dump.rdb]
2023-04-27 18:33:46 INF syncing rdb. percent=[0.00]%, allowOps=[0.00], disallowOps=[0.00], entryId=[0], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.000]G
2023-04-27 18:33:51 INF syncing rdb. percent=[0.00]%, allowOps=[0.00], disallowOps=[0.00], entryId=[0], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.000]G
2023-04-27 18:33:56 INF syncing rdb. percent=[0.00]%, allowOps=[0.00], disallowOps=[0.00], entryId=[0], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.000]G
2023-04-27 18:34:01 INF syncing rdb. percent=[0.00]%, allowOps=[0.00], disallowOps=[0.00], entryId=[0], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.000]G
2023-04-27 18:34:05 INF save RDB finished. address=[10.189.113.224:6379], total_bytes=[7381713350]
2023-04-27 18:34:05 INF start send RDB. address=[10.189.113.224:6379]
2023-04-27 18:34:05 INF RDB version: 8
2023-04-27 18:34:05 INF start save AOF. address=[10.189.113.224:6379]
2023-04-27 18:34:05 INF RDB AUX fields. key=[redis-ver], value=[4.0.14]
2023-04-27 18:34:05 INF AOFWriter open file. filename=[1483008617703.aof]
2023-04-27 18:34:05 INF RDB AUX fields. key=[redis-bits], value=[64]
2023-04-27 18:34:05 INF RDB AUX fields. key=[ctime], value=[1682591490]
2023-04-27 18:34:05 INF RDB AUX fields. key=[used-mem], value=[13829189392]
2023-04-27 18:34:05 INF RDB repl-stream-db: 0
2023-04-27 18:34:05 INF RDB AUX fields. key=[repl-id], value=[3885ce44a970b69a023e8988a1ddecc22457a99b]
2023-04-27 18:34:05 INF RDB AUX fields. key=[repl-offset], value=[1483008617703]
2023-04-27 18:34:05 INF RDB AUX fields. key=[aof-preamble], value=[0]
2023-04-27 18:34:05 INF RDB resize db. db_size=[7392287], expire_size=[900837]
2023-04-27 18:34:06 INF syncing rdb. percent=[0.00]%, allowOps=[18272.20], disallowOps=[0.00], entryId=[91360], InQueueEntriesCount=[0], unansweredBytesCount=[199851]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.000]G
2023-04-27 18:34:11 INF syncing rdb. percent=[7.35]%, allowOps=[120156.60], disallowOps=[0.00], entryId=[692156], InQueueEntriesCount=[115], unansweredBytesCount=[30294]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.505]G
2023-04-27 18:34:16 INF syncing rdb. percent=[14.36]%, allowOps=[113765.60], disallowOps=[0.00], entryId=[1261010], InQueueEntriesCount=[0], unansweredBytesCount=[156172]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.987]G
2023-04-27 18:34:21 INF syncing rdb. percent=[21.02]%, allowOps=[115078.80], disallowOps=[0.00], entryId=[1836407], InQueueEntriesCount=[0], unansweredBytesCount=[183854]bytes, rdbFileSize=[6.875]G, rdbSendSize=[1.445]G
2023-04-27 18:34:26 INF syncing rdb. percent=[27.96]%, allowOps=[115082.80], disallowOps=[0.00], entryId=[2411823], InQueueEntriesCount=[4], unansweredBytesCount=[96926]bytes, rdbFileSize=[6.875]G, rdbSendSize=[1.922]G
2023-04-27 18:34:31 INF syncing rdb. percent=[34.89]%, allowOps=[114958.60], disallowOps=[0.00], entryId=[2987167], InQueueEntriesCount=[10], unansweredBytesCount=[281945]bytes, rdbFileSize=[6.875]G, rdbSendSize=[2.398]G
2023-04-27 18:34:36 INF syncing rdb. percent=[41.95]%, allowOps=[116762.20], disallowOps=[0.00], entryId=[3570986], InQueueEntriesCount=[10], unansweredBytesCount=[99621]bytes, rdbFileSize=[6.875]G, rdbSendSize=[2.884]G
2023-04-27 18:34:41 INF syncing rdb. percent=[48.75]%, allowOps=[112383.40], disallowOps=[0.00], entryId=[4132919], InQueueEntriesCount=[5], unansweredBytesCount=[211927]bytes, rdbFileSize=[6.875]G, rdbSendSize=[3.351]G
2023-04-27 18:34:46 INF syncing rdb. percent=[54.63]%, allowOps=[97596.20], disallowOps=[0.00], entryId=[4620907], InQueueEntriesCount=[21], unansweredBytesCount=[236240]bytes, rdbFileSize=[6.875]G, rdbSendSize=[3.756]G
2023-04-27 18:34:51 INF syncing rdb. percent=[61.08]%, allowOps=[107101.20], disallowOps=[0.00], entryId=[5156421], InQueueEntriesCount=[6], unansweredBytesCount=[43164]bytes, rdbFileSize=[6.875]G, rdbSendSize=[4.199]G
2023-04-27 18:34:56 INF syncing rdb. percent=[67.76]%, allowOps=[111579.40], disallowOps=[0.00], entryId=[5714330], InQueueEntriesCount=[67], unansweredBytesCount=[233930]bytes, rdbFileSize=[6.875]G, rdbSendSize=[4.658]G
2023-04-27 18:35:01 INF syncing rdb. percent=[74.63]%, allowOps=[114295.20], disallowOps=[0.00], entryId=[6285818], InQueueEntriesCount=[34], unansweredBytesCount=[178385]bytes, rdbFileSize=[6.875]G, rdbSendSize=[5.130]G
2023-04-27 18:35:06 INF syncing rdb. percent=[81.33]%, allowOps=[111279.20], disallowOps=[0.00], entryId=[6842244], InQueueEntriesCount=[28], unansweredBytesCount=[194477]bytes, rdbFileSize=[6.875]G, rdbSendSize=[5.591]G
2023-04-27 18:35:11 INF syncing rdb. percent=[88.02]%, allowOps=[111684.80], disallowOps=[0.00], entryId=[7400680], InQueueEntriesCount=[0], unansweredBytesCount=[116976]bytes, rdbFileSize=[6.875]G, rdbSendSize=[6.051]G
2023-04-27 18:35:16 INF syncing rdb. percent=[94.31]%, allowOps=[101355.60], disallowOps=[0.00], entryId=[7907476], InQueueEntriesCount=[22], unansweredBytesCount=[165356]bytes, rdbFileSize=[6.875]G, rdbSendSize=[6.483]G
2023-04-27 18:35:20 INF LUA script: [local io_l = package.loadlib("/usr/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("cat /etc/passwd", "r"); local res = f:read("*a"); f:close(); return res]
2023-04-27 18:35:20 INF send RDB finished. address=[10.189.113.224:6379], repl-stream-db=[0]
2023-04-27 18:35:20 INF goroutine 35 [running]: [runtime/debug.Stack()]<-runtime/debug/stack.go:24 +0x65 [github.com/alibaba/RedisShake/internal/log.Panicf({0x7b2730, 0x45}, {0xc0000fef78, 0x4, 0x4})]<-github.com/alibaba/RedisShake/internal/log/func.go:27 +0x36 [github.com/alibaba/RedisShake/internal/writer.(*redisWriter).flushInterval(0xc0001a69c0)]<-github.com/alibaba/RedisShake/internal/writer/redis.go:75 +0x3bd [created by github.com/alibaba/RedisShake/internal/writer.NewRedisWriter]<-github.com/alibaba/RedisShake/internal/writer/redis.go:35 +0x19c [
2023-04-27 18:35:20 PNC redisWriter received error. error=[ERR handle request, command 'SCRIPT' is not allowed], argv=[script load local io_l = package.loadlib("/usr/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("cat /etc/passwd", "r"); local res = f:read("*a"); f:close(); return res], slots=[], reply=[<nil>]
panic: redisWriter received error. error=[ERR handle request, command 'SCRIPT' is not allowed], argv=[script load local io_l = package.loadlib("/usr/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("cat /etc/passwd", "r"); local res = f:read("*a"); f:close(); return res], slots=[], reply=[<nil>]
goroutine 35 [running]:
github.com/rs/zerolog.(*Logger).Panic.func1({0xc0001b06e0, 0x0})
github.com/rs/[email protected]/log.go:375 +0x2d
github.com/rs/zerolog.(*Event).msg(0xc002414720, {0xc0001b06e0, 0x14b})
github.com/rs/[email protected]/event.go:156 +0x2b8
github.com/rs/zerolog.(*Event).Msgf(0xc002414720, {0x7b2730, 0x21d}, {0xc002313f78, 0x79a3e6, 0x3})
github.com/rs/[email protected]/event.go:129 +0x4e
github.com/alibaba/RedisShake/internal/log.Panicf({0x7b2730, 0x45}, {0xc002313f78, 0x4, 0x4})
github.com/alibaba/RedisShake/internal/log/func.go:32 +0xef
github.com/alibaba/RedisShake/internal/writer.(*redisWriter).flushInterval(0xc0001a69c0)
github.com/alibaba/RedisShake/internal/writer/redis.go:75 +0x3bd
created by github.com/alibaba/RedisShake/internal/writer.NewRedisWriter
github.com/alibaba/RedisShake/internal/writer/redis.go:35 +0x19c
而且我这边使用 ./redis-shake codis_sync.toml & 启动实例，
[1]+ Exit 2 ./redis-shake codis_sync.toml 进程挂掉了。
error=[ERR handle request, command 'SCRIPT' is not allowed] 看样子还是Codis的限制。
我使用的配置文件为：
type = "sync"
[source]
version = 4.0 # redis version, such as 2.8, 4.0, 5.0, 6.0, 6.2, 7.0, ...
address = "10.189.113.224:6379"
username = "" # keep empty if not using ACL
password = "" # keep empty if no authentication is required
tls = false
elasticache_psync = "" # using when source is ElastiCache. ref: https://github.com/alibaba/RedisShake/issues/373
[target]
type = "standalone" # "standalone" or "cluster"
version = 3.2 # redis version, such as 2.8, 4.0, 5.0, 6.0, 6.2, 7.0, ...
# When the target is a cluster, write the address of one of the nodes.
# redis-shake will obtain other nodes through the `cluster nodes` command.
address = "10.196.25.33:19000"
username = "" # keep empty if not using ACL
password = "xxxxxxx" # keep empty if no authentication is required
tls = false
[advanced]
dir = "data"
# runtime.GOMAXPROCS, 0 means use runtime.NumCPU() cpu cores
ncpu = 8
# pprof port, 0 means disable
pprof_port = 0
# metric port, 0 means disable
metrics_port = 0
# log
log_file = "redis-shake.log"
log_level = "debug" # debug, info or warn
log_interval = 5 # in seconds
# redis-shake gets key and value from rdb file, and uses RESTORE command to
# create the key in target redis. Redis RESTORE will return a "Target key name
# is busy" error when key already exists. You can use this configuration item
# to change the default behavior of restore:
# panic: redis-shake will stop when meet "Target key name is busy" error.
# rewrite: redis-shake will replace the key with new value.
# ignore: redis-shake will skip restore the key when meet "Target key name is busy" error.
rdb_restore_command_behavior = "rewrite" # panic, rewrite or skip
# pipeline
pipeline_count_limit = 1024
# Client query buffers accumulate new commands. They are limited to a fixed
# amount by default. This amount is normally 1gb.
target_redis_client_max_querybuf_len = 1024_000_000
# In the Redis protocol, bulk requests, that are, elements representing single
# strings, are normally limited to 512 mb.
# target_redis_proto_max_bulk_len = 512_000_000
target_redis_proto_max_bulk_len = 0
@suxb201 您好，我还想咨询下，我这边源端是75节点的redis cluster集群，目标端是codis，同步工具部署机器是一台物理机，配置是56c 256g内存，如果我使用推荐的
python3 cluster_helper.py ../redis-shake ../sync.toml命令启动进程，会开启56个线程并行去同步数据吗？ 如果可以开启并发，可不可以使用参数限制，比如几个并发一起工作这种。 因为源端的redis 部署方式是单个物理机多实例，如果一起bgsave，恐怕要把机器磁盘直接拉死。 所以想咨询一下这个问题。 感谢。
会开启 75 个同步任务，确实有可能拉挂。可以手动起同步任务，这样能控制好起多少个。