ptubes icon indicating copy to clipboard operation
ptubes copied to clipboard

[Feature]It is recommended that ptubes support the separation of storage and computation

Open myksl opened this issue 3 years ago • 0 comments

Describe the feature

I recently encountered a problem. Our business has some tasks with relatively large traffic, which will write a large amount of data. At the same time, because of business characteristics, we expect the data to be stored for a long time, which will cause our disk usage to be relatively high. In extreme cases will cause disk exhaustion So, I hope to have a new ability

  1. Added a new StorageMode inside Reader, which supports S3 remote storage + Memory mode, and supports tiered storage
  2. Support Binlog persistent storage
  3. The real-time consumption stream maintains the same speed as before
  4. The local disk will be used as the real-time data layer, S3 will be used as the persistent storage layer, and the non-latest data of the local disk will be asynchronously written to S3 through certain rules

myksl avatar May 20 '22 08:05 myksl