tidb
tidb copied to clipboard
pingcap
workload-learning: Extract metrics from cluster statements stats
What problem does this PR solve?
Issue Number: close #58131
Problem Summary:
What changed and how does it work?
Combines the workload analyze function with the statements stats snapshot function.
- Extract the original workload data from the snapshot of statements stats
- Extract the memory and time consumed by the table in this query based on the query execution record and type of each operator
- Index (Pan index)
- PointGet (Pan Point get)
- TableReader (Pan)
- IndexMerge(Pan)
- Calculate the total memory and time of each table: group and accumulate the results of the second step based on the table id
- Calculate the total memory and time of all tables: total scan time, total mem usage
- Calculate the table cost ratio of each table based on the table cost formula.
table cost = table scan time / total scan time + table mem usage / total mem usage
Check List
Tests
Adding Tests
- [ ] Unit test
- [ ] Integration test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test
- [ ] I checked and no code files have been changed.
Side effects
- [ ] Performance regression: Consumes more CPU
- [ ] Performance regression: Consumes more Memory
- [ ] Breaking backward compatibility
Documentation
- [ ] Affects user behaviors
- [ ] Contains syntax changes
- [ ] Contains variable changes
- [ ] Contains experimental features
- [ ] Changes MySQL compatibility
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.
None
Hi @elsa0520. Thanks for your PR.
PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.
I understand the commands that are listed here.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
![tiprow[bot] avatar](https://avatars.githubusercontent.com/in/18927?v=4 "Published by a pub.dev verified publisher"){kind=small}
@qw4990: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.
In response to this:
/retest
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@elsa0520: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.
In response to this:
/test build
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
Codecov Report
Attention: Patch coverage is 6.47321% with 419 lines in your changes missing coverage. Please review.
Project coverage is 74.7658%. Comparing base (
6a12f31) to head (f7ce1c2). Report is 39 commits behind head on master.
Additional details and impacted files
@@ Coverage Diff @@
## master #61378 +/- ##
================================================
+ Coverage 73.0660% 74.7658% +1.6998%
================================================
Files 1730 1782 +52
Lines 481492 497746 +16254
================================================
+ Hits 351807 372144 +20337
+ Misses 108156 102465 -5691
- Partials 21529 23137 +1608
| Flag | Coverage Δ | |
|---|---|---|
| integration | 48.8718% <0.6696%> (?) |
|
| unit | 72.3450% <6.4732%> (+0.0442%) |
:arrow_up: |
Flags with carried forward coverage won't be shown. Click here to find out more.
| Components | Coverage Δ | |
|---|---|---|
| dumpling | 52.7804% <ø> (ø) |
|
| parser | ∅ <ø> (∅) |
|
| br | 62.2422% <ø> (+15.5550%) |
:arrow_up: |
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
- :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
[LGTM Timeline notifier]
Timeline:
2025-06-05 03:53:27.096040367 +0000 UTC m=+500293.298385629: :ballot_box_with_check: agreed by qw4990.2025-06-20 06:31:40.083451819 +0000 UTC m=+426152.806630801: :ballot_box_with_check: agreed by AilinKid.
![ti-chi-bot[bot] avatar](https://avatars.githubusercontent.com/in/214286?v=4 "Published by a pub.dev verified publisher"){kind=small}
@elsa0520: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:
/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-integration-e2e-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test
The following commands are available to trigger optional jobs:
/test pingcap/tidb/canary_ghpr_unit_test
/test pull-br-integration-test-next-gen
/test pull-build-next-gen
/test pull-check-next-gen
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-ddl-test-next-gen
/test pull-integration-e2e-test-next-gen
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-integration-realcluster-test-next-gen
/test pull-mysql-client-test-next-gen
/test pull-mysql-test-next-gen
/test pull-sqllogic-test
/test pull-tiflash-integration-test
/test pull-unit-test-ddlv1-next-gen
/test pull-unit-test-next-gen
Use /test all to run the following jobs that were automatically triggered:
pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_integration_e2e_test
pingcap/tidb/pull_mysql_client_test
In response to this:
/test fast_test_tiprow
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@elsa0520: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.
In response to this:
/test fast_test_tiprow
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@ti-chi-bot[bot]: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.
In response to this:
@elsa0520: The specified target(s) for
/testwere not found. The following commands are available to trigger required jobs:/test build/test check-dev/test check-dev2/test mysql-test/test pull-br-integration-test/test pull-integration-ddl-test/test pull-integration-e2e-test/test pull-lightning-integration-test/test pull-mysql-client-test/test pull-unit-test-ddlv1/test unit-testThe following commands are available to trigger optional jobs:
/test pingcap/tidb/canary_ghpr_unit_test/test pull-br-integration-test-next-gen/test pull-build-next-gen/test pull-check-next-gen/test pull-common-test/test pull-e2e-test/test pull-integration-common-test/test pull-integration-copr-test/test pull-integration-ddl-test-next-gen/test pull-integration-e2e-test-next-gen/test pull-integration-jdbc-test/test pull-integration-mysql-test/test pull-integration-nodejs-test/test pull-integration-python-orm-test/test pull-integration-realcluster-test-next-gen/test pull-mysql-client-test-next-gen/test pull-mysql-test-next-gen/test pull-sqllogic-test/test pull-tiflash-integration-test/test pull-unit-test-ddlv1-next-gen/test pull-unit-test-next-genUse
/test allto run the following jobs that were automatically triggered:pingcap/tidb/ghpr_buildpingcap/tidb/ghpr_checkpingcap/tidb/ghpr_check2pingcap/tidb/ghpr_mysql_testpingcap/tidb/ghpr_unit_testpingcap/tidb/pull_integration_ddl_testpingcap/tidb/pull_integration_e2e_testpingcap/tidb/pull_mysql_client_testIn response to this:
/test fast_test_tiprow
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: AilinKid, hawkingrei, qw4990
The full list of commands accepted by this bot can be found here.
The pull request process is described here
- ~~OWNERS~~ [AilinKid,hawkingrei,qw4990]
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
