feat: add support to record performance report of database
Related #723
Required agent PR https://github.com/frappe/agent/pull/129
Changes -
- Added
fetch_performance_reportmethod inDatabase Serverdoctype - Added
Performance Reportdoctype which will contain performance schema reports - Added all the reports (available in mysql workbench) as child table of
Performance Report - Anyone can enable/disable specific performance report from `Database Server > Mariadb Settings"
- Added cronjob to fetch reports every 15 minutes
Preview
https://github.com/frappe/press/assets/57363826/29ff4bed-484f-4f3e-aedd-0079d419dc63
Codecov Report
Attention: Patch coverage is 3.34928% with 202 lines in your changes are missing coverage. Please review.
Project coverage is 39.50%. Comparing base (
3e2f95b) to head (7ad47d9). Report is 32 commits behind head on master.
:exclamation: Current head 7ad47d9 differs from pull request most recent head 559984b
Please upload reports for the commit 559984b to get more accurate results.
Additional details and impacted files
@@ Coverage Diff @@
## master #1395 +/- ##
==========================================
- Coverage 39.73% 39.50% -0.23%
==========================================
Files 331 347 +16
Lines 26996 25096 -1900
==========================================
- Hits 10726 9915 -811
+ Misses 16270 15181 -1089
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
@tanmoysrt awesome stuff!
I've some queries and suggestions.
- How much time this takes to fetch from DB and roughly what's size of data per day (assuming 15 min interval)
- Does this capture
events_statemens_summary_by_digestor equivalent digest summaries? (found it to be most useful replacement for slow query logs) - If you implement summary by digest you'll need to keep truncating the stats and regenerate it on our end using historical data. The digest has some hard limit (10K queries) after which it stops capturing details and stores them against
NULLdigest. - Can we setup log clearing so this doctype and all child tables get cleared at certain interval. Ref: https://github.com/frappe/frappe/pull/17159 see how it's done for email queue. We will need fast clearing in raw SQL without touching ORM.
- It takes 5~6 seconds to fetch from DB and then couple of seconds to fetch from server to press end. Although it may vary for large database as I was testing on just development setup. Although if we start checking Innodb buffer stats, it can slowdown perf schema query + normal queries according to docs
- We are not collecting
events_statements_summary_by_digestbut we can add that. just a little change required. Check this PR, you will find out the sql statements of available reports, that we are fetching from DB - Need to check
- Yup, we can setup log clearing, will check the PR.
cc @ankush