tiflow icon indicating copy to clipboard operation
tiflow copied to clipboard

TiCDC table span ResolvedTs advancing can be improved

Open hicqu opened this issue 1 year ago • 0 comments

Before asking a question, make sure you have

  • [X] Searched existing Stack Overflow questions
  • [X] Googled your question
  • [X] Searched open and closed GitHub issues
  • [X] Read the documentation

What is your question?

Currently, TiCDC uses cdc/puller.Frontier to advance ResolvedTs. It's based on a heap with some other complex data structures like skiplist.

For a given number set, heap is good for such situation: after every time we update one number in the set, we need to calculate top(k) for them. However, For advancing ResolvedTs, we just need to get top(1) after a duration, and there could be million updates in the duration. So heap won't make helpful in this case, just iterate all numbers in the set is fine enough.

hicqu avatar Jan 31 '24 13:01 hicqu