pika icon indicating copy to clipboard operation
pika copied to clipboard

feat: floyd supports one key for one data structure

Open Mixficsol opened this issue 1 year ago • 5 comments

背景

当前 Pika 的一个 Key 可以对应多种数据结构,和 Redis 不一致

解决方案

Floyd 现有的设计之上,将之前 String 类型所在的Column-Family 用于存放所有的 KeyMeta 信息

修改前:
enum ColumnFamilyIndex {
    kStringsCF = 0,
    kHashesMetaCF = 1,
    kHashesDataCF = 2,
    kSetsMetaCF = 3,
    kSetsDataCF = 4,
    kListsMetaCF = 5,
    kListsDataCF = 6,
    kZsetsMetaCF = 7,
    kZsetsDataCF = 8,
    kZsetsScoreCF = 9,
    kStreamsMetaCF = 10,
    kStreamsDataCF = 11,
};

修改后:
enum ColumnFamilyIndex {
    kMetaCF = 0,
    kHashesDataCF = 1,
    kSetsDataCF = 2,
    kListsDataCF = 3,
    kZsetsDataCF = 4,
    kZsetsScoreCF = 5,
    kStreamsDataCF = 6,
};

kMetaCF 字段设计

修改前

String

key格式
| reserve1 | key | reserve2 | 
|    8B    |     |   16B    |

value格式
| value | reserve | cdate | timestamp | 
|       |   16B   |   8B  |    8B     |

Hash

key格式
| reserve1 | key | reserve2 | 
|    8B    |     |    16B   |

value格式
| hash_size | version |  reserve  | cdate | timestamp | 
|    4B     |    8B   |    16B    |   8B  |     8B    |

List

key格式
| reserve1 | key | reserve2 |
|    8B    |     |   16B    |

value格式
|list_size| version | left index | right index | reserve |  cdate | timestamp | 
|   8B    |   8B    |     8B     |     8B      |   16B   |   8B   |    8B     |

set

key格式
| reserve1 | key | reserve2 |
|    8B    |     |    16B    |

value格式
| set_size | version | reserve | cdate | timestamp | 
|    4B    |    8B   |   16B   |   8B  |    8B     |

zset

key格式
| reserve1 | key | reserve2 |
|    8B    |     |    16B   |  

value格式
| zset_size | version | reserve  | cdate | timestamp | 
|    4B     |    8B   |    16B   |   8B  |    8B     |

Stream

key格式
| reserve1 | key | reserve2 |
|    8B    |     |    16B   |  

value格式
|  group_id_ | entries_added_ | first_id_ms | first_id_seq | last_id_ms | last_id_seq | max_deleted_entry_ms | max_deleted_entry_seq | length | version |
|     4B     |       8B       |     8B      |      8B      |     8B     |      8B     |         8B           |          8B           |   4B   |    4B   |


修改后

我们对每一种数据类型的 Metavalue 前增加一个字段 Type 用于区别每个 Key 对应的数据结构

Set 类型举例

key格式
| reserve1 | key | reserve2 |
|    8B    |     |    16B   |

value格式
| type | set_size | version | reserve | cdate | timestamp | 
|  1B  |    4B    |    8B   |   16B   |   8B  |    8B     |

String 的 Meta 格式

key格式
| reserve1 | key | reserve2 | 
|    8B    |     |   16B    |

value格式
| Type | value | reserve | cdate | timestamp | 
|  1B  |       |   16B   |  8B   |    8B     |

在解析的时候,先解析头部的第一个字节,然后根据类型判断是否需要继续解析下去

Mixficsol avatar Apr 17 '24 08:04 Mixficsol

  1. Cmd 层多加了一次对返回值的判断(用于判断多 key 的错误信息)
else if (s_.ToString() == ErrTypeMessage) {
  res_.SetRes(CmdRes::kMultiKey);
} 
  1. Cache 层删除了 Redis 前缀
// 修改前:
std::string CachePrefixKeyK = PCacheKeyPrefixK + key_;
db_->cache()->SetBitIfKeyExist(CachePrefixKeyK, bit_offset_, on_);

// 修改后
db_->cache()->SetBitIfKeyExist(key_, bit_offset_, on_);
  1. Storage 层对 Expire, Del, Exists, Expireat, Persist, TTL, GetType 这几个之前需要遍历每种数据类型的接口修改成指定操作某一个具体类型的接口
修改前:
// Keys Commands
int32_t Storage::Expire(const Slice& key, int64_t ttl, std::map<DataType, Status>* type_status) {
  type_status->clear();
  int32_t ret = 0;
  bool is_corruption = false;

  auto& inst = GetDBInstance(key);
  // Strings
  Status s = inst->StringsExpire(key, ttl);
  if (s.ok()) {
    ret++;
  } else if (!s.IsNotFound()) {
    is_corruption = true;
    (*type_status)[DataType::kStrings] = s;
  }

  // Hash
  s = inst->HashesExpire(key, ttl);
  if (s.ok()) {
    ret++;
  } else if (!s.IsNotFound()) {
    is_corruption = true;
    (*type_status)[DataType::kHashes] = s;
  }

  // Sets
  s = inst->SetsExpire(key, ttl);
  if (s.ok()) {
    ret++;
  } else if (!s.IsNotFound()) {
    is_corruption = true;
    (*type_status)[DataType::kSets] = s;
  }

  // Lists
  s = inst->ListsExpire(key, ttl);
  if (s.ok()) {
    ret++;
  } else if (!s.IsNotFound()) {
    is_corruption = true;
    (*type_status)[DataType::kLists] = s;
  }

  // Zsets
  s = inst->ZsetsExpire(key, ttl);
  if (s.ok()) {
    ret++;
  } else if (!s.IsNotFound()) {
    is_corruption = true;
    (*type_status)[DataType::kZSets] = s;
  }

  if (is_corruption) {
    return -1;
  } else {
    return ret;
  }
}

修改后
// Keys Commands
int32_t Storage::Expire(const Slice& key, int64_t ttl) {
  auto& inst = GetDBInstance(key);
  Status s = inst->Expire(key, ttl);
  if (s.ok()) {
    return 1;
  } else if (!s.IsNotFound()) {
    return -1;
  }
  return 1;
}
  1. Redis 层每次需要对 Key 进行修改时,新增了对 meta-key 类型的判断
if (!ExpectedMetaValue(Type::kSet, meta_value)) {
  return Status::InvalidArgument("WRONGTYPE Operation against a key holding the wrong kind of value");
}

Mixficsol avatar Apr 25 '24 10:04 Mixficsol

todo: slot migrate 支持 stream

AlexStocks avatar May 07 '24 11:05 AlexStocks

Bot detected the issue body's language is not English, translate it automatically.


todo: slot migrate support stream

Issues-translate-bot avatar May 07 '24 11:05 Issues-translate-bot

补充下新架构的图片吧,在 shaoyi 的图片基础之上进行修改

AlexStocks avatar May 13 '24 04:05 AlexStocks

Bot detected the issue body's language is not English, translate it automatically.


Let’s add pictures of the new architecture and modify them based on shaoyi’s pictures.

Issues-translate-bot avatar May 13 '24 04:05 Issues-translate-bot