incr caculate error with ttl
Is this a regression?
Yes
Description
1 主节点连续执行:setex key1 10 value1, (value1=100), incr key1。主节点这边的key1会变成101, 且10s后就会过期 2 假设从节点30s后才收到binlog,分别是一条pksetat key1 ....和incur key1。从节点的这条pksetat采用了主的过期时间戳,所以key1一写进去就是过期的,相当于没写这个key1。尔后执行incur key1, 就制造出了一个永远不会过期的key1。这就导致了主从不一致了。
Please provide a link to a minimal reproduction of the bug
No response
Screenshots or videos
解决办法: 和鸽姐讨论了一下,鸽姐提了一个很好的方案,而且不影响兼容: incr也写binlog的时候也用pksetat来写就行,具体地: 执行incr的时候,如果发现目标是个有ttl且还没过期的key, 将其ttl取出来,构造一个pksetat Binlog发给从消费。 举例: 1 主节点执行: 1.1 setex key1 10 1 (10s后过期,value为1) 1.2 incur key1 2 在生成incr的Binlog时构造一个pksetat Binlog,从节点Apply这个Binlog的时候相当于set key1 11 (并且给了一个和主那边一样的过期时间戳)
这样的话至少incr引起的这个问题解决了。
总的来说工作量会比较大。
1 有问题的命令不止incr,需要 1.2 梳理哪些结构支持过期时间
1.3 有哪些命令有incr这种性质(会给你无中生有)
2 刚刚发现,setex的binlog重写为了pksetat命令,但是setnx这种一样性质的命令却没有重写Binlog,这块也需要检查,所有类似性质的命令都需要check一下
Please provide the version you discovered this bug in (check about page for version information)
No response
Anything else?
No response
详细梳理及方案
String 类型的:
SET key value: 如果键不存在,则创建该键。
INCR key: 如果键不存在,则创建该键,初始值为 0,然后递增。
APPEND key value: 如果键不存在,则创建该键并设置值为 value。
Hash数据类型的:
HSET key field value: 如果键不存在,则创建该键,并添加哈希字段和值。
HINCRBY key field increment: 如果键不存在,则创建该键,初始哈希字段值为 0,然后进行递增操作。
List数据类型的:
LPUSH key value [value ...]: 如果键不存在,则创建一个新列表,并将 value 插入到列表头部。(先不修复)
RPUSH key value [value ...]: 如果键不存在,则创建一个新列表,并将 value 插入到列表尾部。(先不修复)
Set数据类型的:
SADD key member [member ...]: 如果键不存在,则创建一个新集合,并添加成员。(先不修复)
SMOVE source destination member: 如果目标键不存在,则创建一个新集合,并将成员移动到这个新集合中。(先不修复)
Sorted Set(有序集合)数据类型:
ZADD key score member [score member ...]: 如果键不存在,则创建一个新有序集合,并添加成员和对应的分值。(先不修复)
上述这些命令会在操作过程中检测到不存在的key时候进行自动创建,从而实现无中生有
需要修改的地方:
sting命令
1.Incr命令在从节点写binlog的时候同时需要将ttl传过去
2.Apend命令在从节点写binlog的时候将该key的ttl传过去
Hash 命令
1.HINCRBY key field increme 命令在给从节点写binlog时将ttl携带上
List数据类型
LPUSH key value [value ...] 在传输的过程中将该key的ttl携带上
目前先只修改 incr 和append命令即可 hash需要待支持feild和value单独过期后做出来
Bot detected the issue body's language is not English, translate it automatically.
Detailed review and plan
String type:
SET key value: If the key does not exist, create the key.
INCR key: If the key does not exist, create the key with an initial value of 0 and then increment.
APPEND key value: If the key does not exist, create the key and set the value to value.
Hash data type:
HSET key field value: If the key does not exist, create the key and add the hash field and value.
HINCRBY key field increment: If the key does not exist, create the key with an initial hash field value of 0 and then increment it.
List data type:
LPUSH key value [value ...]: If the key does not exist, create a new list and insert value at the head of the list. (don’t fix it yet)
RPUSH key value [value ...]: If the key does not exist, create a new list and insert value at the end of the list. (don’t fix it yet)
Set data type:
SADD key member [member ...]: If the key does not exist, create a new collection and add the member. (don’t fix it yet)
SMOVE source destination member: If the target key does not exist, create a new collection and move the member to this new collection. (don’t fix it yet)
**Sorted Set (ordered set) data type: **
ZADD key score member [score member ...]: If the key does not exist, create a new ordered set and add members and corresponding scores. (don’t fix it yet)
The above commands will automatically create them when a non-existent key is detected during the operation, thus creating something out of nothing.
What needs to be modified:
sting command
-
The Incr command needs to pass the ttl when writing the binlog from the node.
-
The Apend command passes the ttl of the key when writing binlog from the slave node.
Hash command
- The HINCRBY key field increme command carries ttl when writing binlog to the slave node.
List data type
LPUSH key value [value ...] carries the ttl of the key during the transmission process
For now, just modify the incr and hincr commands.
cjg:incr coding;hincr todo
Bot detected the issue body's language is not English, translate it automatically.
cjg:incr coding;hincr all
incr,hincr, string, hash @车金鸽 本周改完
Bot detected the issue body's language is not English, translate it automatically.
incr,hincr, string, hash @Chejinge Completed the changes this week