ptubes icon indicating copy to clipboard operation
ptubes copied to clipboard

[Feature]Hope ptubes can support storage layer partition

Open myksl opened this issue 3 years ago • 0 comments

Describe the feature

At present, the concept of sharding is a logical concept, which only exists in the Client for parallel consumption. As a result, each Reader needs to pull the full binlog, and there is a visible performance upper limit for pulling and storing (cannot scale linearly)

  1. From logical sharding to physical sharding, the concept of partition is introduced, each Reader does not need to pull all binlogs, reducing storage and processing pressure, and at the same time supporting parallel processing of data by Reader (each Reader processes a part of the partition), improving ptubes Scalability
  2. The data of each partition can be subscribed separately to reduce the data backtracking when the service is duplicated
  3. Each partition has a separate writing thread, effectively breaking through the performance bottleneck of single-threaded sequential writing

myksl avatar May 20 '22 08:05 myksl