flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[tidb]fix the tidb checkpoints to exactly once.

Open ArainChen opened this issue 1 year ago • 4 comments

#2034 I fix that bug.

ArainChen avatar Mar 23 '23 06:03 ArainChen

I used the same method, but there is another problem: when starting from checkpoint, I get an exception occasionally 捕获

524687460 avatar Jun 12 '23 07:06 524687460

I used the same method, but there is another problem: when starting from checkpoint, I get an exception occasionally 捕获

[why problem appear] I guess this is because the data in tikv was dropped by gc. Therefore, the offset data stored in checkpoints can no longer be found in tikv. If the flink program recovers quickly from checkpoints and the tikv memory is not dropped by gc, no exceptions appear. In contrast, if flink recovers slowly from checkpoints and the tikv memory is dropped by gc, an exception will occur. [how to fix it] I think the solution should be to extend the gc interval of tikv.

ArainChen avatar Jun 13 '23 14:06 ArainChen

Thanks @ArainChen for the great work! Before this PR could be merged, could you please rebase it with latest master branch?

yuxiqian avatar Apr 26 '24 02:04 yuxiqian

This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs.

github-actions[bot] avatar Jul 17 '24 00:07 github-actions[bot]