tiflow icon indicating copy to clipboard operation
tiflow copied to clipboard

TiKV initial scan task not cancel/termicated when changefeed paused or failed

Open fubinzh opened this issue 4 months ago • 8 comments

What did you do?

  1. 07:51 - 07:55 Create 4 CDC clusters (each CDC cluster with 32C CDC nodes), for each CDC cluster create one changefeed, each changefeed replicating ~40k tables (overall ~160k tables)
  2. 07:58 Pause changefeed and update config
  3. 08:12 resume all changefeeds
  4. 09:38 Pause all changefeed
  5. 10:33 delete changefeed (Delete changefeeds from TiDB cloud UI)
  6. 10:07 - 10:33 manually kill and restart all TiKV

What did you expect to see?

  • When task paused, previous pending initial scan task should be cancelled, Otherwise this might block following changefeed initilization.

What did you see instead?

After step 1, one changefeed initilization finished (all tables added the changefeed) After step 3, For the initilized changefeed, table count is 0 for 1.4h+ before it was paused. After step 4, initial scan pending tasks is decreasing, but it is very slow. After step 6, the pending tasks disappeared.

image image

Versions of the cluster

v6.5.9

fubinzh avatar Oct 09 '24 06:10 fubinzh