ddl: fast reorg of index may cause duplicate entry error

[AilinKid]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

Look, we did add a unique index with ddl job id as 1131, while at 02.19 ddl owner has changed, and resume the checkpoint at same time, consequently leading duplicate entry error at 02.21

3. What did you see instead (Required)

no duplicate error (since user has checked with their data)

4. What is your TiDB version? (Required)

master, 7.1.1

[lance6716]

Same type of root cause like https://github.com/pingcap/tidb/issues/46986

[lance6716]

after https://github.com/pingcap/tidb/pull/52993 , there are still 2 cases can trigger the false alert of "duplicate entry"

• non distributed execution framework: due to concurrency DDL worker and low watermark checkpoint, some data will be imported twice with different TS when DDL owner changes
• distributed execution framework with local sort: there's no checkpoint manager so TS is allocated when needed and does not persist

cc @tangenta

[kennedy8312]

Are there any pending tasks for this issue? Shall we close this one?

[lance6716]

Are there any pending tasks for this issue? Shall we close this one?

It's still not totally fixed, I have listed 2 cases in above comments. It's hard to fix, maybe we can fix it in v8.3