add index is always running after injecting pd leader io delay 1s last for 1mins and then recover

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、tidb_enable_dist_task='on' 2、run sysbench 3、inject pd leader io delay 1s last for 1mins

logs: tidb-0.tar.gz tidb-1.tar.gz

2. What did you expect to see? (Required)

add index can success

3. What did you see instead (Required)

add index is always running after fault recover

the status of ddl job is not synced after 1h0m0s (now: 2024-02-21 02:27:46, jobId: 554, job type: add index /* ingest /, state: running) operatorLogs: [2024-02-21 01:27:20] ###### start adding index alter table sbtest1 add index index_test_1708450040691 (c) [2024-02-21 01:27:20] ###### wait for ddl job finish [2024-02-21 02:27:46] ###### wait for ddl job finish timeout(1h0m0s) select job_id, job_type, state from information_schema.ddl_jobs where query = 'alter table sbtest1 add index index_test_1708450040691 (c)' jobId: 554, job type: add index / ingest */, state: running"

tidb log always print： [2024/02/21 02:26:56.039 +08:00] [INFO] [reorg.go:264] ["run reorg job wait timeout"] [category=ddl] ["wait time"=5s] ["total added row count"=70008818] [2024/02/21 02:26:56.040 +08:00] [INFO] [ddl_worker.go:1381] ["schema version doesn't change"] [category=ddl] [jobID=554]

4. What is your TiDB version? (Required)

./tidb-server -V Release Version: v8.0.0-alpha Edition: Community Git Commit Hash: aca145b164905ebe48f03c46a54843234e2c99ba Git Branch: heads/refs/tags/v8.0.0-alpha UTC Build Time: 2024-02-18 12:22:27 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore 2024-02-21T01:05:50.694+0800

[Lily2025]

/assign ywqzzy /type bug /severity major

[ywqzzy]

Can't reproduce it again