RedisShake
RedisShake copied to clipboard
tair-opensource
常见问题已看，但没有解决。错误信息：PNC EOF， INF AOFWriter close file
问题描述 迁移过程中遇到如下错误, 常见问题中说是源端的缓存太小了，实际看了下源端，还挺大的。 我们在将腾讯云redis的数据迁移到Ucloud redis,之前方案预研时，复制了腾讯云的线上实例是可以迁移的，结果今天真正线上执行时就发现报错了。数据量也并没有变大。
config get client-output-buffer-limit "client-output-buffer-limit" "normal 0 0 0 slave 4347483648 4347483647 60 pubsub 33554432 8388608 60"
INF goroutine 6 [running]: [runtime/debug.Stack()]<-runtime/debug/stack.go:24 +0x65 [github.com/alibaba/RedisShake/internal/log.Panicf({0x79a4ac, 0x3}, {0x0, 0x0, 0x0})]<-github.com/alibaba/RedisShake/internal/log/func.go:27 +0x36 [github.com/alibaba/RedisShake/internal/log.PanicError({0x8167a0, 0xc000094060})]<-github.com/alibaba/RedisShake/internal/log/func.go:36 +0x33 [github.com/alibaba/RedisShake/internal/reader.(*psyncReader).saveAOF(0xc000088640, {0x816540, 0xc0000560c0})]<-github.com/alibaba/RedisShake/internal/reader/psync.go:202 +0x166 [created by github.com/alibaba/RedisShake/internal/reader.(*psyncReader).StartRead.func1]<-github.com/alibaba/RedisShake/internal/reader/psync.go:48 +0xef [ 2023-05-24 09:48:20 PNC EOF 2023-05-24 09:48:20 INF AOFWriter close file. filename=[2134954018571.aof], filesize=[5418393]
redis-shake 的日志：
{"level":"info","time":"2023-05-24T09:47:48+03:00","message":"syncing aof. allowOps=[113.80], disallowOps=[0.00], entryId=[1224870], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[209], aofReceivedOffset=[2134958273964], aofAppliedOffset=[2134958273755]"}
{"level":"info","time":"2023-05-24T09:47:53+03:00","message":"syncing aof. allowOps=[105.80], disallowOps=[0.00], entryId=[1225401], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[198], aofReceivedOffset=[2134958479861], aofAppliedOffset=[2134958479663]"}
{"level":"info","time":"2023-05-24T09:47:58+03:00","message":"syncing aof. allowOps=[117.20], disallowOps=[0.00], entryId=[1225987], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[2134958848088], aofAppliedOffset=[2134958848088]"}
{"level":"info","time":"2023-05-24T09:48:03+03:00","message":"syncing aof. allowOps=[91.40], disallowOps=[0.00], entryId=[1226444], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[2134959062502], aofAppliedOffset=[2134959062502]"}
{"level":"info","time":"2023-05-24T09:48:08+03:00","message":"syncing aof. allowOps=[89.60], disallowOps=[0.00], entryId=[1226892], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[248], aofReceivedOffset=[2134959212678], aofAppliedOffset=[2134959212430]"}
{"level":"info","time":"2023-05-24T09:48:13+03:00","message":"syncing aof. allowOps=[83.80], disallowOps=[0.00], entryId=[1227311], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[2134959307023], aofAppliedOffset=[2134959307023]"}
{"level":"info","time":"2023-05-24T09:48:18+03:00","message":"syncing aof. allowOps=[92.20], disallowOps=[0.00], entryId=[1227772], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[2134959418928], aofAppliedOffset=[2134959418928]"}
{"level":"info","time":"2023-05-24T09:48:20+03:00","message":"goroutine 6 [running]: [runtime/debug.Stack()]<-runtime/debug/stack.go:24 +0x65 [github.com/alibaba/RedisShake/internal/log.Panicf({0x79a4ac, 0x3}, {0x0, 0x0, 0x0})]<-github.com/alibaba/RedisShake/internal/log/func.go:27 +0x36 [github.com/alibaba/RedisShake/internal/log.PanicError({0x8167a0, 0xc000094060})]<-github.com/alibaba/RedisShake/internal/log/func.go:36 +0x33 [github.com/alibaba/RedisShake/internal/reader.(*psyncReader).saveAOF(0xc000088640, {0x816540, 0xc0000560c0})]<-github.com/alibaba/RedisShake/internal/reader/psync.go:202 +0x166 [created by github.com/alibaba/RedisShake/internal/reader.(*psyncReader).StartRead.func1]<-github.com/alibaba/RedisShake/internal/reader/psync.go:48 +0xef ["}
{"level":"panic","time":"2023-05-24T09:48:20+03:00","message":"EOF"}
{"level":"info","time":"2023-05-24T09:48:20+03:00","message":"AOFWriter close file. filename=[2134954018571.aof], filesize=[5418393]"}
源端 Redis 版本：云厂商，腾讯 5.0 日志：暂时没有
日志贴在这里
目的端 Redis 版本：UCloud 日志：
你好，我们用的3.1.10这个版本也报的这个错误。
2023-05-24 06:59:52 INF syncing aof. allowOps=[111.60], disallowOps=[0.00], entryId=[1239113], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[104], aofReceivedOffset=[2134528929512], aofAppliedOffset=[2134528929408]
2023-05-24 06:59:53 PNC EOF
2023-05-24 06:59:53 INF AOFWriter close file. filename=[2134523603347.aof], filesize=[5356119]
panic: EOF
goroutine 8 [running]:
github.com/rs/zerolog.(*Logger).Panic.func1({0xc0008a3032, 0x0})
github.com/rs/[email protected]/log.go:359 +0x2d
github.com/rs/zerolog.(*Event).msg(0xc000076960, {0xc0008a3032, 0x3})
github.com/rs/[email protected]/event.go:156 +0x2b8
github.com/rs/zerolog.(*Event).Msg(...)
github.com/rs/[email protected]/event.go:108
github.com/alibaba/RedisShake/internal/log.logFinally(0xc000076960, {0x79f1e9, 0xc0000100d8}, {0x0, 0x1, 0xc00003ff08})
github.com/alibaba/RedisShake/internal/log/func.go:77 +0x53
github.com/alibaba/RedisShake/internal/log.Panicf({0x79f1e9, 0x3}, {0x0, 0x0, 0x0})
github.com/alibaba/RedisShake/internal/log/func.go:27 +0x57
github.com/alibaba/RedisShake/internal/log.PanicError({0x81b8a0, 0xc000020080})
github.com/alibaba/RedisShake/internal/log/func.go:31 +0x33
github.com/alibaba/RedisShake/internal/reader.(*psyncReader).saveAOF(0xc000088050, {0x81b640, 0xc000076660})
github.com/alibaba/RedisShake/internal/reader/psync.go:202 +0x166
created by github.com/alibaba/RedisShake/internal/reader.(*psyncReader).StartRead.func1
github.com/alibaba/RedisShake/internal/reader/psync.go:48 +0xef
看了下常见问题，说是可能是源端的缓存太小了，实际看了下源端，还挺大的 config get client-output-buffer-limit
"client-output-buffer-limit" "normal 0 0 0 slave 4347483648 4347483647 60 pubsub 33554432 8388608 60"
请问这个错误还有其他原因吗，怎么解决，谢谢
@571255945 看代码依旧是腾讯云的 Redis 给断开的连接，可以发工单咨询下腾讯云工程师。另问，请问你这是同步多久后被断开的？
@suxb201 你好，我迁移过程中我发现，InQueueEntriesCount这个一直是0，之前预研测试时都是有数值的，今天线上运行一直是0，这个也不正常吧？
2023-05-24 12:56:36 INF send RDB finished. address=[10.230.9.12:6379], repl-stream-db=[0] 2023-05-24 12:56:37 INF AOFReader open file. aof_filename=[535729131647.aof] 2023-05-24 12:56:40 INF syncing aof. allowOps=[14774.60], disallowOps=[0.00], entryId=[196728], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[79], aofReceivedOffset=[535729263377], aofAppliedOffset=[535729263298] 2023-05-24 12:56:45 INF syncing aof. allowOps=[170.20], disallowOps=[0.00], entryId=[197579], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[535729316942], aofAppliedOffset=[535729316942] 2023-05-24 12:56:50 INF syncing aof. allowOps=[141.80], disallowOps=[0.00], entryId=[198288], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[535729349634], aofAppliedOffset=[535729349634] 2023-05-24 12:56:55 INF syncing aof. allowOps=[130.40], disallowOps=[0.00], entryId=[198940], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[29], aofReceivedOffset=[535729383736], aofAppliedOffset=[535729383707] 2023-05-24 12:57:00 INF syncing aof. allowOps=[169.60], disallowOps=[0.00], entryId=[199788], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[29], aofReceivedOffset=[535733395570], aofAppliedOffset=[535733395541] 2023-05-24 12:57:05 INF syncing aof. allowOps=[180.60], disallowOps=[0.00], entryId=[200691], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[535733446746], aofAppliedOffset=[535733446746] 2023-05-24 12:57:10 INF syncing aof. allowOps=[163.00], disallowOps=[0.00], entryId=[201506], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[29], aofReceivedOffset=[535733492744], aofAppliedOffset=[535733492715] 2023-05-24 12:57:15 INF syncing aof. allowOps=[166.00], disallowOps=[0.00], entryId=[202336], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[29], aofReceivedOffset=[535733546283], aofAppliedOffset=[535733546254] 2023-05-24 12:57:20 INF syncing aof. allowOps=[193.80], disallowOps=[0.00], entryId=[203305], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[535733607369], aofAppliedOffset=[535733607369] 2023-05-24 12:57:25 INF syncing aof. allowOps=[209.40], disallowOps=[0.00], entryId=[204352], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[535733683584], aofAppliedOffset=[535733683584] 2023-05-24 12:57:30 INF syncing aof. allowOps=[161.40], disallowOps=[0.00], entryId=[205159], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, diff=[0], aofReceivedOffset=[535733727279], aofAppliedOffset=[535733727279]
@suxb201 你好，我们联系了腾讯，之前这个参数之前设置的是2分钟，我们这边测试大概2分钟左右就断了，后来腾讯那边把超时时间改为10分钟，然后我们这边测试大概10分钟就又断了。 我的理解，原理上redis-shake工具应该当做从节点加入到了源redis集群中，所以这个从节点的“心跳”，应该是redis-shake来发的吧，所以如果真的没有数据发给腾讯，是不是就是redis-shake的问题？
@571255945 redis-shake 每经过 100ms 会汇报一次点位。 你可以抓包确认下 shake 确实回复了点位。 腾讯的 Redis 经过了 Proxy，怀疑 Proxy 没收了点位汇报命令。