automq
automq copied to clipboard
Scale-in nodes failover compact the StreamSetObjects
Who is this for and what problem do they have today?
Why is solving this problem impactful?
Compact the StreamSetObjects of the scale-in nodes with splitThreshold=1MiB :
- Most of the data: Most of the data is split into StreamObjects to achieve independent TTL and better data cohesiveness.
- Remaining 5000 partitions * 1MB: The remaining data will be periodically scanned and eventually expire through StreamControlManager#cleanupScaleInNodes.
Additional notes
We need to prepare more CPU / Memory resources for compaction failover.
https://github.com/AutoMQ/automq-for-rocketmq/tree/kafka_issues706
Implement it when you have a more elegant solution.