kyuubi icon indicating copy to clipboard operation
kyuubi copied to clipboard

[KYUUBI #6107] Collect and summarize the `executorRunTime` and `executorCpuTime` of the statement

Open XorSum opened this issue 11 months ago • 1 comments

:mag: Description

Issue References 🔗

This pull request fixes #6107

Describe Your Solution 🔧

The total execution time of a statement (or a session) is the summary of the execution time of the stages belonging to the statement (or session). The total execution time of a stage is collected from SQLOperationListener#onStageCompleted. The total execution times of the statement or a session are stored in the engine events or output to the log.

Types of changes :bookmark:

  • [ ] Bugfix (non-breaking change which fixes an issue)
  • [x] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request :coffin:

Behavior With This Pull Request :tada:

Related Unit Tests


Checklist 📝

Be nice. Be informative.

XorSum avatar Feb 28 '24 08:02 XorSum

Codecov Report

Attention: Patch coverage is 97.87234% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 61.13%. Comparing base (84e60e9) to head (d9efa2d). Report is 1 commits behind head on master.

Files Patch % Lines
...org/apache/spark/kyuubi/SQLOperationListener.scala 90.90% 0 Missing and 1 partial :warning:
Additional details and impacted files
@@            Coverage Diff            @@
##             master    #6112   +/-   ##
=========================================
  Coverage     61.12%   61.13%           
  Complexity       23       23           
=========================================
  Files           623      624    +1     
  Lines         37195    37237   +42     
  Branches       5039     5041    +2     
=========================================
+ Hits          22737    22766   +29     
- Misses        12005    12007    +2     
- Partials       2453     2464   +11     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Feb 28 '24 10:02 codecov-commenter

LGTM overall. Is there any screenshot or logging output to show the results?

截屏2024-02-29 11 07 30 截屏2024-02-29 11 07 44

XorSum avatar Feb 29 '24 03:02 XorSum

LGTM. Thank you for your contribution. Can you update the screenshot to the description of the PR?

done

XorSum avatar Feb 29 '24 06:02 XorSum

Thanks. Could you have another try for replacing formatDurationVerbose with formatDuration, which is in a shorter form for logging? Sorry to brother you again.

bowenliang123 avatar Feb 29 '24 06:02 bowenliang123

Thanks. Could you have another try for replacing formatDurationVerbose with formatDuration, which is in a shorter form for logging? Sorry to brother you again.

done

XorSum avatar Feb 29 '24 06:02 XorSum

Thanks, merged to master.

cxzl25 avatar Mar 04 '24 03:03 cxzl25