Make EventCacheCount in go-mysql configurable
Is your feature request related to a problem?
The current DM-Worker memory consumption profile shows that a significant portion of the Go Heap (∼20 GiB in our tests) is consumed by large transaction buffers, primarily within the go-mysql replication loop (specifically, the (*RowsEvent).decodeImage function).
While the Go runtime's allocation is involved, the primary constraint is the size and retention of the event objects created during binlog decoding. The EventCacheCount variable, currently hardcoded, controls the size of the event object pool (EventCache).
In high-throughput environments dealing with very large transactions (e.g., 50 million row INSERTs or large schema migrations), setting a high hardcoded EventCacheCount means the DM-Worker pre-allocates and retains a substantial amount of memory for event objects, which unnecessarily contributes to the Resident Set Size (RSS) and can trigger Out-of-Memory (OOM) issues on resource-constrained hosts.
Describe the feature you'd like
I propose exposing the EventCacheCount field (defined in go-mysql/replication/binlogsyncer.go#L129) as a configurable parameter that DM users can set in their task configuration. https://github.com/go-mysql-org/go-mysql/blob/master/replication/binlogsyncer.go#L129
This will allow users to:
- Reduce Memory Consumption: Lower the EventCacheCount value in environments where memory is critical, thus reducing the size of the pre-allocated event object pool.
- Optimize Performance: Increase the value if memory is plentiful and the application frequently processes large, complex events, reducing the overhead of repeated memory allocation.
Describe alternatives you've considered
No response
Teachability, Documentation, Adoption, Migration Strategy
No response