beam
beam copied to clipboard
apache
[TPC-DS] Store metrics into BigQuery and InfluxDB
Store TPC-DS metrics into BigQuery and InfluxDB:
- Collect the results and send them into BigQuery and InfluxDB for every benchmark job;
- Related Jenkins jobs are adjusted;
- Removed Spark 2 runner support (Spark 2 is deprecated in Beam);
- Added Grafana dashboards files;
- Added more TPC-DS queries, that should pass with Beam SQL, to run on Jenkins.
Fixes #21254
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Choose reviewer(s) and mention them in a comment (
R: @username). - [ ] Mention the appropriate issue in your description (for example:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead. - [ ] Update
CHANGES.mdwith noteworthy changes. - [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This PR is blocked on #22260 to address the rest of review comments regarding the new InfluxDB tool API.
@mosche I updated PR with your recent InfluxDBPublisher changes. PTAL