Occasional performance regression on benchmarksql

[owlsome2501]

Bug Report

There is occasional performance regression on benchmarksql. releated issue: #55426

1. Minimal reproduce step (Required)

1. Deploy a cluster with 3 TiDB and 3 Tikv.
2. Run benchmark benchmarksql (benchmarksql-1k) multiple times until find the specific io or cpu usage pattern on grafana.

2. What did you expect to see? (Required)

No performance regression. QPS: 64039 (+6.4% compared to baseline 60193) or QPS: 61259 (+1.77% compared to baseline 60193)

3. What did you see instead (Required)

Performance regression . QPS: 55187 (-8.3% compared to baseline 60193) or QPS: 54639 (-9.20% compared to baseline 60193)

4. What is your TiDB version? (Required)

pingcap/tidb@26443dab89ba5b1a866de3c3e44e2f51468bc435 tikv/tikv@7c612787d0bfc57a339907ebcb26777dc296fd5f tikv/pd@098b802fcda75b5ae0527b0b50b95ba8b31403cf or pingcap/tidb@74034d4ac243b3c14dbf5f8a9edb92e740da4212 tikv/tikv@de06b38a118582ee01d16ab4d3582bd8fa5cb635 tikv/pd@76bc34aee22554353bd7ea9785d3772dcb7ab31b

[owlsome2501]

/severity major /label affects-8.3 /label affects-8.4

[ti-chi-bot[bot]]

@owlsome2501: The label(s) affects-8.3 cannot be applied. These labels are supported: fuzz/sqlancer, challenge-program, compatibility-breaker, first-time-contributor, contribution, good first issue, correctness, duplicate, proposal, security, needs-more-info, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, needs-cherry-pick-release-8.1, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.4, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5, may-affects-8.1.

In response to this:

/severity major /label affects-8.3 /label affects-8.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[jebter]

@owlsome2501 Can you confirm which modules introduced the issue?

[owlsome2501]

Unable to determine the module. It may be hot spots or load balancing problem.