beam
beam copied to clipboard
apache
Backlog metrics do not showing up in FlinkRunner
We have a Flink Job which does not emit backlog metrics. Actually metrics are emitting in KafkaIO. However I could not see them on Flink Metric system. Looks like Beam -> Flink wiring is broken. I set metricContext in Checkpointing phase which is the place metrics emit on UnBoundedReader.
I see previous PR is closed. I recreated against latest master
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead. - [ ] Update
CHANGES.mdwith noteworthy changes. - [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.
See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
There was a significant change in https://github.com/apache/beam/pull/25525 that affects this. The ReaderInvocationUtil#invokeCheckpoint must be used in FlinkUnboundedSourceReader#getReaderCheckpoint as well. For this functionality to be preserved during refactorings, it would be nice to add at least a basic test to verify it works (which would prove it will not - at least for classic runner - in this case, because of the missing use in FlinkUnboundedSourceReader. The UnboudedSourceWrapper should be removed as drafted in https://github.com/apache/beam/pull/31530.
In the original PR there was a suspicion that this causes some issues, is this resolved?
Re-adding my comment from the original PR here since this one is now open.
I am facing a similar issue while using JdbcIO, I can't see the metrics that are emitted in the Flink Metric system. Do the changes here fix the issue, or are there other changes that need to be made for sources/sinks that are not UnboundedSources?
I am using the apache beam version 2.56.0 and beam-runners-flink-1.17
You don't see any metrics, or metrics related to checkpoints?
I am referring specifically to these 2 metrics, the value is always NaN even though data is being ingested in the sink database that I am using
@je-ik Because Metrics are creating when Flink's open function but because they are not getting update at advance time. Flink does not receive updated metrics. My code change make metrics update part of checkpointing time. We are using this fix with 30K+ Flink jobs and build a autoscaling for Beam. https://beam.apache.org/blog/apache-beam-flink-and-kubernetes-part3/
@je-ik Because Metrics are creating when Flink's open function but because they are not getting update at advance time. Flink does not receive updated metrics. My code change make metrics update part of checkpointing time. We are using this fix with 30K+ Flink jobs and build a autoscaling for Beam. https://beam.apache.org/blog/apache-beam-flink-and-kubernetes-part3/
Yes, I understand that. I'm a little struggling why @oswidan97 does not see any metrics in his case. Might be because he is apparently using bounded reader. On the other hand, the patch in this PR affects only portable runner, so I suppose for your autoscaling case you use the portable one, right?
@je-ik Because Metrics are creating when Flink's open function but because they are not getting update at advance time. Flink does not receive updated metrics. My code change make metrics update part of checkpointing time. We are using this fix with 30K+ Flink jobs and build a autoscaling for Beam. https://beam.apache.org/blog/apache-beam-flink-and-kubernetes-part3/
Yes, I understand that. I'm a little struggling why @oswidan97 does not see any metrics in his case. Might be because he is apparently using bounded reader. On the other hand, the patch in this PR affects only portable runner, so I suppose for your autoscaling case you use the portable one, right?
No, I see the confusion. @oswidan97 refers to sink not source. Can you please create different issue for that? This seems unrelated to this issue, because this one is about source metrics.
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.