apache
feat(cluster): add a new command SLOTSIZE
Background
- Issue #2715 and #2723 reported duplicated data during cluster expansion. This PR introduce a way to find out the dirt slots which slot has been migrated but data remained on node, then clear the dirty data on node.
- Users may want to know how many keys in each slot.
Usage
Use command "clusterx slotsize {SlotRange} scan" to scan keys of slot range no node.
Use command "clusterx clearslot {SlotRange}" to clear keys of slot range which slots has been migrated on node.
Use command "clusterx slotsize {SlotRange}" to print out the stats of each slot.
Test Report Env: Tencent SA5.2XLARGE16 CVM Instance(8C16G), 4 CLOUD_TSSD disks each 100GB using 4 stripes. 13.3million keys on node, each slot has an average of 2400 keys. Each slot costs 2-3ms when executing command "slotsize {SlotRange} scan", from min 1ms to max 4ms. No significant variation observed in CPU utilization when scanning slots.
@greatsharp Except for the above comments, I believe we should NOT support the clear subcommand in SLOTSIZE, which makes no sense(also, this command is marked as read-only).
@greatsharp Except for the above comments, I believe we should NOT support the
clearsubcommand inSLOTSIZE, which makes no sense(also, this command is marked as read-only).
removed the clear sub command, we may add the clear dirty logic in 'clusterx migrate' command, or add a new sub command in clusterx command.
@greatsharp I'm generally good with the current implementation, with a few comments. Could you please add Go test cases for those commands? cc @PragmaTwice
@greatsharp I'm generally good with the current implementation, with a few comments. Could you please add Go test cases for those commands? cc @PragmaTwice
sure, wait for a moment.
LGTM, one more comment: GetSlotStats should also only allow to get/scan the slot that belongs to the node. And can clean the old stats if the slot has been moved. cc @PragmaTwice
Quality Gate passed
Issues
0 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
See analysis details on SonarQube Cloud
![sonarqubecloud[bot] avatar](https://avatars.githubusercontent.com/in/12526?v=4 "Published by a pub.dev verified publisher"){kind=small}
LGTM, one more comment:
GetSlotStatsshould also only allow to get/scan the slot that belongs to the node. And can clean the old stats if the slot has been moved. cc @PragmaTwice
How do you know whether there is duplicate dirty data on node or not if you cannot scan the slots which was migrated out?
LGTM, one more comment:
GetSlotStatsshould also only allow to get/scan the slot that belongs to the node. And can clean the old stats if the slot has been moved. cc @PragmaTwiceHow do you know whether there is duplicate dirty data on node or not if you cannot scan the slots which was migrated out?
In my opinion, keep each command clear and accurate
LGTM, one more comment:
GetSlotStatsshould also only allow to get/scan the slot that belongs to the node. And can clean the old stats if the slot has been moved. cc @PragmaTwiceHow do you know whether there is duplicate dirty data on node or not if you cannot scan the slots which was migrated out?
In my opinion, keep each command clear and accurate
You can check this when calling GetSlotStats, it would be easy to know which slots no longer belong to this node.
LGTM, one more comment:
GetSlotStatsshould also only allow to get/scan the slot that belongs to the node. And can clean the old stats if the slot has been moved. cc @PragmaTwiceHow do you know whether there is duplicate dirty data on node or not if you cannot scan the slots which was migrated out?
In my opinion, keep each command clear and accurate
You can check this when calling
GetSlotStats, it would be easy to know which slots no longer belong to this node.
Do you mean we can put the scan and clear process all in one GetSlotStats method? Could you give more details or setsp of it?