feat: Add PacificA data replication consistency scheme

[buzhimingyonghu]

PacificA 协议概述

PacificA 协议简单来说分为两部分：

1. 数据复制
2. 配置管理

由于在 Pika 中，配置管理主要由 pika_sentinel 负责，本文主要关注通过主从模式的数据复制及其与 pika_sentinel 配合的协调过程。

在 Pika 中的应用

主要分为三个部分：

1. PacificA 中主从模式的数据一致流程处理
2. 分布式日志型存储系统的设计
3. 故障恢复后的协调状态

---

启动 PacificA

在 Pika 中，建立普通主从连接的命令为：

slaveof <ip> <port>

如果需要启动 PacificA 协议，需要增加 strong 参数：

slaveof <ip> <port> strong

当从节点执行上述命令时，会触发 slaveofcmd，读取相关参数，并由 pika_server 保存这些信息，随后异步交由 PikaAuxiliaryThread 线程（以下简称 PAT）处理。 PAT 是 PacificA 协议中的核心辅助线程，负责：

- 状态机状态切换
- 主从之间的心跳发送及超时检查
- 主从之间的同步任务

PacificA 主从模式的数据一致流程

主从建立连接的四个阶段

1. MetaSync：主从元数据的同步和检查
2. TrySync：判断数据完整性，选择全量同步或增量同步
3. Candidate：从节点作为候选者，追加完整的准备列表
4. BinlogSync：正式加入集群，开始进行数据复制

下面是基本的数据结构：

MetaSync 阶段

从节点的 PAT 线程通过发送 MetaReq 请求与主节点建立连接，其中包含 is_consistency 字段，表示强一致性请求。 主节点收到请求后，若 consistency 标记为 true，则会：

1. 设置所有数据库的 consistency 标记
2. 初始化上下文
3. 判断是否需要进入协调状态

随后，从节点收到主节点返回的 MetaSyncRes，并执行以下操作：

1. 比较本地和主节点的数据库结构 (db_structs) 是否一致

2. 如果本地 replication_id 与主节点不一致，且本地 replication_id 为空，则执行全量同步；否则进行增量同步

3. 根据同步类型更新从节点的状态：

   • 全量同步：设置状态为 kTryDBSync
   • 增量同步：设置状态为 kTryConnect

TrySync 阶段

全量同步完成后，从节点更新自身的 committedID 和 preparedID，并发送 TrySyncReq 请求，携带 committedID 确认日志一致性。 主节点验证后，返回包含主节点 preparedID 的 TrySyncRes，从节点将 preparedID 与主节点对齐，完成增量同步。

流程总结：

1. 从节点发送 TrySyncReq，带有 committedID。

2. 主节点检查 committedID：

   • 若主节点 committedID 大于从节点，表示同步正常。
   • 若从节点 committedID 大于主节点，表示选主失败。

3. 主节点返回 TrySyncRes，包含主节点的 preparedID，从节点需对齐。

BinlogSync 阶段

主节点收到从节点的第一次 binlog 请求后，将从节点设置为候选者状态，并追加日志。 主节点通过心跳包和 binlog 数据通知从节点，将日志分阶段写入本地：

• 从节点收到 binlogSync 后，先写入 binlog，等待主节点通知哪些请求已提交。
• 主节点收到所有从节点确认后，将请求标记为已提交，更新提交点，确保与所有从节点保持一致。

分布式日志型存储系统的设计

PacificA 中采用逻辑复制的方式，具体包括：

1. 状态的一致性：所有副本逻辑上保持相同的状态，并可处理相同类型的更新和查询。
2. 日志记录：系统在接收到更新请求时，首先将其写入日志中，确保即使系统故障也能通过日志恢复数据。
3. 内存数据结构更新：日志记录完成后，将更新应用到内存中的数据结构中。
4. 定期创建检查点：防止内存溢出，定期将数据快照保存到磁盘，形成持久化检查点。
5. 日志截断：检查点创建后，删除已存储到检查点的日志，优化存储需求。

故障恢复后的协调状态

初始状态

• A 是主节点，B、C 和 D 是副本节点。
• committedB 是 committedA 的子集，committedA 是所有副本 prepared 的子集。

故障恢复

当主节点 A 故障时：

1. 系统重新配置，将 B 提升为新主节点。
2. B 完成协调后，新的 committedB 与旧的 preparedB 保持一致，所有副本的 preparedID 与新主节点的 preparedID 对齐。

一次写请求的操作流程

1. 写 binlog：等待所有从节点追加日志后，执行写入数据库请求。
2. 主节点处理 binlog 请求：
   • 若非一致性模式，按传统主从复制执行。
   • 若为一致性模式：
3. coordinator_ 追加日志并记录 offset。
4. 等待从节点同步，更新主节点的 committedID。
5. 若同步失败（超时 10 秒），记录信息并退出。

PacificA 一致性测试用例说明

测试用例1：基础一致性测试

目的：验证主从复制的基本功能和数据一致性 步骤：

1. 向主节点写入数据
2. 验证两个从节点的数据同步情况
3. 检查所有节点的复制状态 预期结果：所有节点数据完全一致，复制状态正常

测试用例2：并发写入一致性测试

目的：确保并发写入时的数据一致性 步骤：

1. 向主节点并发写入10条数据
2. 等待数据同步完成
3. 验证两个从节点的所有数据 预期结果：所有并发写入的数据都正确同步到从节点

测试用例3：网络分区恢复测试

目的：测试网络分区后的一致性恢复 步骤：

1. 写入初始数据
2. 断开从节点1连接（模拟网络分区）
3. 向主节点写入新数据
4. 恢复从节点1连接
5. 验证数据一致性 预期结果：网络恢复后，断开的从节点应同步所有错过的数据

测试用例4：动态节点添加测试

目的：验证新增节点时的数据一致性 步骤：

1. 初始只启动主节点和一个从节点
2. 写入一批初始数据
3. 添加第二个从节点
4. 写入新的数据
5. 验证新旧数据的一致性 预期结果：新加入的从节点应正确接收所有历史数据和新数据

测试用例5：节点故障恢复测试

目的：测试节点故障和恢复时的系统行为 步骤：

1. 向所有节点写入初始数据
2. 模拟从节点1故障
3. 故障期间写入数据
4. 恢复从节点1
5. 写入新的数据
6. 验证所有数据集 预期结果：

• 节点故障期间系统继续正常运行
• 故障节点恢复后能同步所有错过的数据
• 所有节点最终达到数据一致

测试环境

• 1个主节点（端口：9301）
• 2个从节点（端口：9302，9303）
• 启用强一致性模式

注意事项

1. 每个测试用例都包含足够的等待时间，确保数据同步完成
2. 所有测试都在强一致性模式下进行
3. 测试过程中会验证数据的完整性和一致性

Summary by CodeRabbit

Summary by CodeRabbit

• New Features
  • Introduced a "strong" command option that activates enhanced consistency mode during replication.
  • Expanded log synchronization capabilities with additional parameters for tracking committed and prepared states.
  • Enhanced error reporting with a dedicated consistency timeout response.
  • Upgraded replication protocols by adding new fields to support state and offset tracking.
  • Improved server role and state management to optimize master-slave transitions.
  • Added methods for managing consistency state and log entries in the replication framework.
  • Introduced new member variables to track log offsets and consistency states in various classes.
  • Added integration tests for verifying consistency in master-slave replication scenarios.
  • Automated cleanup script for Pika and PacificA instances.
  • Enhanced replication consistency checks and error handling in various components.
  • Added new constants for PacificA consistency testing configuration.
  • Enhanced acknowledgment handling during binlog synchronization.
  • Updated methods for handling prepared IDs and consistency checks in replication processes.

[coderabbitai[bot]]

[!IMPORTANT]

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

This pull request refines the system’s consistency management and log offset handling. It introduces new member variables, method overloads, and API modifications that extend internal state tracking for replication and synchronization. Several classes across administration, consensus, replication, and server components receive enhancements to support consistency flags, improved error handling, and more detailed log operations. Additionally, protocol buffers are updated to carry consistency-related identifiers. Overall, the changes add new methods and modify control flow to better manage consistency in distributed operations without altering the public interface.

Changes

File(s)	Change Summary
include/pika_admin.h	Added private member is_consistency_cmd_ in SlaveofCmd and updated the Clear method to reset related variables.
include/pika_binlog.h	Added an overloaded Put method accepting a string, log offset pointer, and binlog output; updated existing Put signature with additional parameters (LogOffset* and bool is_consistency).
include/pika_command.h	Added new enum value kConsistencyTimeout in CmdRes::CmdRet and updated the message() method to handle consistency timeout errors.
include/pika_consensus.h	Introduced new Log class with log management methods; added multiple new public methods in ConsensusCoordinator for appending, truncating, and committing binlogs; and introduced new member variables for consistency and committed IDs.
include/pika_define.h	Added enumerator KCandidate to the SlaveState enum and updated the WriteTask struct with a new member committed_id_ and an associated constructor.
include/pika_rm.h	Added several new public methods in SyncMasterDB (e.g., ConsensusLastIndex(), SetConsistency(), AppendSlaveEntries(), etc.) and in PikaReplicaManager (added BuildBinlogOffset()), enhancing log offset handling and coordination functions.
include/pika_server.h	Added new member variables last_role_ and is_consistency_; updated SetMaster signature to include a consistency flag; and added new methods (last_role(), IsConsistency(), SetConsistency()) to track server role and consistency state.
include/pika_slave_node.h	Updated sent_offset and acked_offset to have default initializations and added a new member target_offset for improved offset tracking.
src/pika_admin.cc	Extended SlaveofCmd command handling in DoInitial() to support a new "strong" argument and updated the SetMaster() call to include the consistency flag.
src/pika_binlog.cc	Added a new Put method for handling string parameters with binlog encoding and updated the existing Put method to include additional parameters for log offsets and consistency checks, ensuring thread-safe operations with a DEFER mechanism.
src/pika_command.cc	Modified InternalProcessCommand to change the control flow based on the server's consistency state—invoking DoBinlog() and DoCommand() in different orders and enhancing error handling for binlog timeouts in consistency mode.
src/pika_consensus.cc	Added numerous new methods in ConsensusCoordinator (e.g., SetConsistency(), AppendEntries(), CommitAppLog(), etc.), enhanced log management via the new Log class, and implemented functions to append, truncate, and search log entries.
src/pika_db.cc	Modified DB::TryUpdateMasterOffset to set and log the master database’s prepared and committed IDs after updating the master offset.
src/pika_inner_message.proto	Extended the protocol buffer definitions by adding optional fields: is_consistency in MetaSync, committed_id in TrySync (request), and prepared_id plus committed_id in the corresponding response messages, thereby enhancing state tracking in synchronization messages.
src/pika_repl_bgworker.cc	Added a new variable ack_end and updated binlog write handling to check for database consistency, conditionally committing the application log based on the consistency state.
src/pika_repl_client.cc	Enhanced SendMetaSync() and SendTrySync() methods to include consistency checks and populate the committed_id field when the master database is in a consistent state.
src/pika_repl_client_conn.cc	Modified HandleTrySyncResponse to check and update the database’s prepared ID against the incoming prepared_id, triggering log truncation if required under consistency conditions.
src/pika_repl_server.cc	Updated BuildBinlogSyncResp to include the committed_id field in responses when the server is in a consistency state.
src/pika_repl_server_conn.cc	Enhanced replication connection methods by incorporating consistency state checks in handling MetaSync, TrySync, and BinlogSync requests, and improved error responses when there is a mismatch in committed offsets.
src/pika_rm.cc	Added new methods in SyncMasterDB (e.g., checkFinished(), SetConsistency(), UpdateCommittedID(), etc.) and in PikaReplicaManager (added BuildBinlogOffset()); refined logic in methods like SyncBinlogToWq, WakeUpSlaveBinlogSync, and ConsensusProposeLog to include consistency checks and timeout mechanisms.
src/pika_server.cc	Extended PikaServer with a new method last_role(), updated SetMaster() to accept a consistency flag, and added IsConsistency() and SetConsistency() methods, along with updates in role management during master/slave transitions.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Cmd
    participant Server
    Client->>Cmd: Send command
    Cmd->>Server: Check IsConsistency()
    alt Consistent Mode
        Server-->>Cmd: returns true
        Cmd->>Cmd: Call DoBinlog()
        Cmd->>Cmd: Evaluate binlog result
        Cmd->>Cmd: Call DoCommand()
    else Inconsistent Mode
        Server-->>Cmd: returns false
        Cmd->>Cmd: Call DoCommand()
        Cmd->>Cmd: Call DoBinlog()
    end
    Cmd->>Client: Return response

sequenceDiagram
    participant AdminCmd
    participant SlaveofCmd
    participant PikaServer
    AdminCmd->>SlaveofCmd: Execute DoInitial("strong")
    SlaveofCmd->>SlaveofCmd: Set is_consistency_cmd_ = true
    SlaveofCmd->>PikaServer: Call SetMaster(master_ip, master_port, true)
    PikaServer->>PikaServer: Update is_consistency_ and role info
    PikaServer-->>SlaveofCmd: Acknowledge master setting
    SlaveofCmd->>AdminCmd: Return command response

Suggested reviewers

• AlexStocks
• Mixficsol
• dingxiaoshuai123

Poem

I'm a coding bunny, hopping through the log,
Tracking offsets and flags like a frog.
Consistency rises—a flag set so true,
In every method, there's magic anew.
With hops and bytes, I celebrate the day,
A whimsical debug in a rabbit's ballet!
🐇💻 Hop on to the code display!

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[buzhimingyonghu]

[PacificA一致性协议概述]：https://cv9vkf72u6.feishu.cn/docx/SSUodyFEJoOWnzxZiJ2cFSLGn8T?from=from_copylink