ptubes
ptubes copied to clipboard
[Feature]Hope ptubes can support storage layer partition
Describe the feature
At present, the concept of sharding is a logical concept, which only exists in the Client for parallel consumption. As a result, each Reader needs to pull the full binlog, and there is a visible performance upper limit for pulling and storing (cannot scale linearly)
- From logical sharding to physical sharding, the concept of partition is introduced, each Reader does not need to pull all binlogs, reducing storage and processing pressure, and at the same time supporting parallel processing of data by Reader (each Reader processes a part of the partition), improving ptubes Scalability
- The data of each partition can be subscribed separately to reduce the data backtracking when the service is duplicated
- Each partition has a separate writing thread, effectively breaking through the performance bottleneck of single-threaded sequential writing