OpenAtomFoundation
feat: migrate tools support pika v3.5.0
feature
pika-migrate-tools支持3.5.0。
Test
写入SET/HSET/LPUSH/SADD/ZADD 各10000000条，然后运行migrate-tool。在迁移期间写入/DEL/HDEL/LPOP/SREM/ZREM各5000000条。在sourceDB和targetDB对操作的key进行一致性测试，结果均一致。
[!IMPORTANT]
Review skipped
Auto reviews are disabled on base/target branches other than the default branch.
Please check the settings in the CodeRabbit UI or the
.coderabbit.yamlfile in this repository. To trigger a single review, invoke the@coderabbitai reviewcommand.You can disable this status message by setting the
reviews.review_statustofalsein the CodeRabbit configuration file.
✨ Finishing Touches
🧪 Generate unit tests
- [ ] Create PR with unit tests
- [ ] Post copyable unit tests in a comment
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands and usage tips.
![coderabbitai[bot] avatar](https://avatars.githubusercontent.com/in/347564?v=4 "Published by a pub.dev verified publisher"){kind=small}
Bot detected the issue body's language is not English, translate it automatically.
I just found out that your changes seem to be under pika/src or pika/include. Shouldn't you modify pika/tools/migrate_tool?
刚发现，你的改动好像是在pika/src或者是pika/include下，是不应该修改 pika/tools/migrate_tool下呢？ 3.5当时直接在src下面改了，我不知道重新commit会不会导致你以前的review记录没了，所以想着review结束了再和4.0一样放到tools下面。当时unstable分支下面有这个工具，3.5没有，我就直接在src下面改了。
Bot detected the issue body's language is not English, translate it automatically.
I just found out that your changes seem to be under pika/src or pika/include. Shouldn't you modify pika/tools/migrate_tool? 3.5 was changed directly under src at that time. I don’t know if recommit will cause your previous review record to be lost, so I thought that after the review ended, I put it under tools like 4.0. At that time, there was this tool under the unstable branch, but it didn't have it in 3.5, so I changed it directly under src.
Bot detected the issue body's language is not English, translate it automatically.
Here I tried to use the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream
这里我尝试使用该工具进行迁移，发现其仅将sst文件下载到migrate-tool本地，但是没有将命令转发给下游
@guozhihao-224 可以贴一下pika_migrate用的conf吗？还有使用场景是啥样的呢？
Bot detected the issue body's language is not English, translate it automatically.
Here I tried using the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream
@guozhihao-224 Can you post the conf used for pika_migrate? What are the usage scenarios?
这里我尝试使用该工具进行迁移，发现其仅将 sst 文件下载到 migrate-tool 本地，但是没有将命令转发给下游
@guozhihao-224可以贴一下pika_migrate用的conf吗？还有使用场景是啥样的呢？ 是pika3.5.5 迁移到 3.5.5版本，100GB量级的数据
这里测试后，发现有将命令往下转发给下游，但是我发现，设置redis-sender-num大于8的话，就会在转发一部分数据后卡住不动了，设置成其他值会的比如5 — 8 之间，也会突然出现卡住不执行的状态。有试过设置为2能够跑完迁移任务，但是设置为2后，迁移时间太长了，100GB的数据的迁移1个多小时
是否有死锁的情况出现 对应的conf： pika-migrate.txt
Bot detected the issue body's language is not English, translate it automatically.
Here I tried using the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream
@guozhihao-224 Can you post the conf used by pika_migrate? What are the usage scenarios? After testing here, I found that the command is forwarded downstream to the downstream, but I found that if the redis-sender-num is greater than 8, it will be stuck after forwarding part of the data, but there is no problem with setting 8. This may be due to the influence of that direction setting.
Corresponding conf: pika-migrate.txt
这里我尝试使用该工具进行迁移，发现其仅将 sst 文件下载到 migrate-tool 本地，但是没有将命令转发给下游
@guozhihao-224可以贴一下pika_migrate用的conf吗？还有使用场景是啥样的呢？ 是pika3.5.5 迁移到 3.5.5版本，100GB量级的数据
这里测试后，发现有将命令往下转发给下游，但是我发现，设置redis-sender-num大于8的话，就会在转发一部分数据后卡住不动了，设置成其他值会的比如5 — 8 之间，也会突然出现卡住不执行的状态。有试过设置为2能够跑完迁移任务，但是设置为2后，迁移时间太长了，100GB的数据的迁移1个多小时
是否有死锁的情况出现 对应的conf： pika-migrate.txt
这边可能还需要提供一下pika_migrate的日志，我这边测试5-8之间同量级的数据均没有出现卡住的情况。
Bot detected the issue body's language is not English, translate it automatically.
Here I tried using the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream
@guozhihao-224 Can you post the conf used by pika_migrate? What are the usage scenarios? is pika3.5.5 migrated to 3.5.5 version, 100GB data
After testing here, I found that the command is forwarded downstream to the downstream, but I found that if the redis-sender-num is greater than 8, it will be stuck after forwarding part of the data. If the setting to other values, such as between 5 and 8, it will suddenly become stuck and not executed. I have tried setting it to 2 to complete the migration task, but after setting it to 2, the migration time is too long, and the migration of 100GB of data will take more than an hour.
Whether there is a deadlock? Corresponding conf: pika-migrate.txt
Here, you may also need to provide the log of pika_migrate. I tested that the data of the same magnitude between 5-8 did not get stuck.
Bot detected the issue body's language is not English, translate it automatically.
After creating a new branch, there will be no so many file modifications and conflicts in pr. After the function is completed, compile the binary and put it in migrate_tool. A new branch has been created.