Update BigQuerySinkMetrics for StreamingInserts.
Update the following metrics after we insert a batch of rows using BigQuery's InsertAll RPC.
1. RowsAppendedCount Counter
Tracks the status of BigQuery Rows after the batch of InsertAll RPCs is completed.
Metric has labels the following labels:
RowStatus: Status of the BigQuery rows after , one ofSUCCESSFUL,RETRIED,FAILEDRpcStatus: Rpc StatusTableId: 'datasets/{ }/tables/{ }' that the rows are sent to.
Rows are updated based on the following criteria.
- Rows that are successfully inserted will have their
RowStatusset to successful. - Rows that have to be retried because the RPC failed will have their
RowStatusset toRETRIEDandRpcStatusset to the failed RPC's status. - Rows might fail to be inserted even when the
InsertAllRPC succeeds. These rows will have theirRowStatusset toRETRIEDandRpcStatusset toINTERNAL. - Rows that fail to be inserted will
RowStatusset toRETRIEDandRpcStatusset toINTERNAL.
2. RpcRequests Counter
Tracks the Rpc Status of the InsertAll Rpc.
Metric has labels the following labels:
Method: BigQuery sink method. Will beSTREAMING_INSERTSfor these metrics.RpcStatus: Rpc StatusTableId: 'datasets/{ }/tables/{ }' that the rows are sent to.
3. RpcLatency Histogram:
Tracks the Rpc Latency of the InsertAll Rpc.
Metric has labels the following labels:
Method: BigQuery sink method. Will beSTREAMING_INSERTSfor these metrics.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead. - [ ] Update
CHANGES.mdwith noteworthy changes. - [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.
R: @m-trieu
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
Thanks for the review Martin, switched StreamingInsertsResults to an AutoValue class, made the instance-variables private and added some update methods to update the instance-variables.