tidb icon indicating copy to clipboard operation
tidb copied to clipboard

slowlog, stmtsummary: include the storage engine(s) a query read from

Open henrybw opened this issue 6 months ago • 5 comments

Issue Number: close #61736

Problem Summary: For deployments involving both TiKV and TiFlash, it's not easy to identify the queries from TiFlash in slow query logs, which makes troubleshooting inconvenient for customers and support engineers. Including extra attributes to identify the storage engine in both the statements summary tables and slow query log would help alleviate this issue.

What changed and how does it work?

  • The IsTiFlash field in the statement context is now set correctly in executorBuilder.buildTableReader() for TiFlash queries. Previously, it was never getting set, because TiFlash queries would early return from this function before getting to the point where IsTiFlash would be set.
  • Add a similar IsTiKV field in the statement context, set at the same time as IsTiFlash, and also set for point get queries.
  • Add storageKV/storageMPP fields to stmtSummaryStats/StmtRecord and populate them with the IsTiKV and IsTiFlash statement context fields, respectively.
  • Add StorageKV/StorageMPP fields to SlowQueryItems and populate them with the IsTiKV and IsTiFlash statement context fields, respectively.
  • Update affected tests for the statement summary and slow query log.
  • Add the new TestStorageEnginesInStmtSummary and TestStorageEnginesInSlowQuery test cases.

Check List

Tests

  • [x] Unit test
  • [x] Integration test
  • [ ] Manual test (add detailed scripts or steps below)
  • [ ] No need to test
    • [ ] I checked and no code files have been changed.

Side effects

  • [ ] Performance regression: Consumes more CPU
  • [ ] Performance regression: Consumes more Memory
  • [ ] Breaking backward compatibility

Documentation

  • [ ] Affects user behaviors
  • [ ] Contains syntax changes
  • [ ] Contains variable changes
  • [ ] Contains experimental features
  • [ ] Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Add `STORAGE_KV` and `STORAGE_MPP` columns to the statement summary tables, and add `Storage_from_kv` and `Storage_from_mpp` columns to the slow query log.

henrybw avatar Jun 13 '25 19:06 henrybw

Hi @henrybw. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow[bot] avatar Jun 13 '25 19:06 tiprow[bot]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 75.5526%. Comparing base (cfe9c12) to head (7475d4a). :warning: Report is 662 commits behind head on master.

Additional details and impacted files
@@               Coverage Diff                @@
##             master     #61737        +/-   ##
================================================
+ Coverage   72.8621%   75.5526%   +2.6905%     
================================================
  Files          1767       1817        +50     
  Lines        486078     502337     +16259     
================================================
+ Hits         354167     379529     +25362     
+ Misses       110250      99603     -10647     
- Partials      21661      23205      +1544     
Flag Coverage Δ
integration 49.4885% <72.7272%> (?)
unit 72.9507% <100.0000%> (+0.8093%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.9532% <ø> (+0.1728%) :arrow_up:
parser ∅ <ø> (∅)
br 63.6797% <ø> (+17.3417%) :arrow_up:
:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Jun 13 '25 19:06 codecov[bot]

/approve

for infoschema pkg

D3Hunter avatar Jun 27 '25 07:06 D3Hunter

/hold

henrybw avatar Jul 23 '25 18:07 henrybw

/unhold

henrybw avatar Jul 25 '25 00:07 henrybw

/retest

henrybw avatar Jul 25 '25 01:07 henrybw

@henrybw: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow[bot] avatar Jul 25 '25 01:07 tiprow[bot]

/ok-to-test

terry1purcell avatar Jul 29 '25 17:07 terry1purcell

/retest

henrybw avatar Jul 29 '25 17:07 henrybw

/lgtm

Tristan1900 avatar Jul 29 '25 19:07 Tristan1900

[LGTM Timeline notifier]

Timeline:

  • 2025-06-17 20:57:35.100250655 +0000 UTC m=+218907.823429636: :ballot_box_with_check: agreed by terry1purcell.
  • 2025-07-29 19:57:19.494105745 +0000 UTC m=+34852.179375991: :ballot_box_with_check: agreed by Tristan1900.

ti-chi-bot[bot] avatar Jul 29 '25 19:07 ti-chi-bot[bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, terry1purcell, Tristan1900

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot[bot] avatar Jul 29 '25 19:07 ti-chi-bot[bot]

@henrybw: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-integration-e2e-test-next-gen 3c572d479133ba5e56d201032b53be8ebcaa8646 link false /test pull-integration-e2e-test-next-gen
non-block/pull-unit-test-next-gen 7475d4a103ba73b1392f597474b71e45506dabc2 link false /test pull-unit-test-next-gen
non-block/pull-mysql-test-next-gen 7475d4a103ba73b1392f597474b71e45506dabc2 link false /test pull-mysql-test-next-gen
non-block/pull-integration-e2e-test-next-gen 7475d4a103ba73b1392f597474b71e45506dabc2 link false /test pull-integration-e2e-test-next-gen

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot[bot] avatar Jul 29 '25 20:07 ti-chi-bot[bot]

/retest-required

terry1purcell avatar Jul 29 '25 21:07 terry1purcell

In response to a cherrypick label: new pull request created to branch release-8.5: #64824. But this PR has conflicts, please resolve them!

ti-chi-bot avatar Dec 03 '25 00:12 ti-chi-bot