tiflow icon indicating copy to clipboard operation
tiflow copied to clipboard

unstable DM-INJECT_DDL_ERROR_SHARDING_BASE2

Open lance6716 opened this issue 1 year ago • 2 comments

Which jobs are flaking?

DM-IT handle_error

Which test(s) are flaking?

DM-INJECT_DDL_ERROR_SHARDING_BASE2

Jenkins logs or GitHub Actions link

[2022-08-03T10:18:27.635Z] [2022/08/03 18:18:24.083 +08:00] [ERROR] [shardddl.go:604] ["error detected when executing SQL job"] [task=test] [unit="binlog replication"] [component=ddl] [error="[code=40061:class=dm-worker:scope=internal:level=high], Message: startLocation: [position: (dm-integration-test-go1180-8555-rn9xh-w2fd0-bin|000001.000001, 9455), gtid-set: 3039ab1a-1315-11ed-abfc-a2c20216ae85:1-38, suffix: 2], endLocation: [position: (dm-integration-test-go1180-8555-rn9xh-w2fd0-bin|000001.000001, 9598), gtid-set: 3039ab1a-1315-11ed-abfc-a2c20216ae85:1-39], origin SQL: [alter table handle_error.tb2 modify column c double]: DDL info for (test, mysql-replica-02) not found, Workaround: Please use show-ddl-locks command to see lock id."]

https://ci2.pingcap.net/blue/organizations/jenkins/dm_ghpr_integration_test/detail/dm_ghpr_integration_test/8555/pipeline

Anything else we need to know

  • Does this test exist for other branches as well?

  • Has there been a high frequency of failure lately?

lance6716 avatar Aug 03 '22 12:08 lance6716

dm-ci.log

lance6716 avatar Aug 03 '22 12:08 lance6716

[2022-08-03T10:18:27.635Z] [2022/08/03 18:18:21.462 +08:00] [WARN] [strategy.go:94] ["retry stratey takes effect"] [error="etcdserver: request timed out"] [retry_times=0] [retry_count=5]

https://github.com/pingcap/tiflow/blob/8577b45f5ba8573157411f5ca7bd26274ae88096/dm/syncer/shardddl/pessimist.go#L138-L139

after retry, since the KV has been deleted by first op, the second delete reports false.

lance6716 avatar Aug 03 '22 13:08 lance6716