automq icon indicating copy to clipboard operation
automq copied to clipboard

Scale-in nodes failover compact the StreamSetObjects

Open superhx opened this issue 1 year ago • 1 comments

Who is this for and what problem do they have today?

Why is solving this problem impactful?

Compact the StreamSetObjects of the scale-in nodes with splitThreshold=1MiB :

  • Most of the data: Most of the data is split into StreamObjects to achieve independent TTL and better data cohesiveness.
  • Remaining 5000 partitions * 1MB: The remaining data will be periodically scanned and eventually expire through StreamControlManager#cleanupScaleInNodes.

Additional notes

superhx avatar Jan 23 '24 07:01 superhx

We need to prepare more CPU / Memory resources for compaction failover.

https://github.com/AutoMQ/automq-for-rocketmq/tree/kafka_issues706

Implement it when you have a more elegant solution.

superhx avatar Jan 23 '24 09:01 superhx