beam icon indicating copy to clipboard operation
beam copied to clipboard

Support poisioning instruction ids to prevent the FnApi data stream from blocking on failed instructions

Open scwhittle opened this issue 1 year ago • 1 comments

Previously it could occur that an instruction id was observed on the control stream but due to exception it would never register a handler and the data stream would be forever blocked trying to multiplex elements to it's queue.

Fixes #32714


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

scwhittle avatar Oct 18 '24 10:10 scwhittle

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

github-actions[bot] avatar Oct 18 '24 11:10 github-actions[bot]

Test failures appear to be unrelated flakiness

scwhittle avatar Oct 21 '24 11:10 scwhittle

assign set of reviewers

scwhittle avatar Oct 21 '24 11:10 scwhittle

Run Java PreCommit

scwhittle avatar Oct 21 '24 11:10 scwhittle

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @robertwb for label java.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

github-actions[bot] avatar Oct 21 '24 11:10 github-actions[bot]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 58.95%. Comparing base (1e27978) to head (abc5492). Report is 121 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #32857      +/-   ##
============================================
+ Coverage     57.41%   58.95%   +1.53%     
- Complexity     1475     3112    +1637     
============================================
  Files           968     1131     +163     
  Lines        154224   174787   +20563     
  Branches       1076     3357    +2281     
============================================
+ Hits          88546   103038   +14492     
- Misses        63477    68398    +4921     
- Partials       2201     3351    +1150     
Flag Coverage Δ
java 69.94% <ø> (+1.35%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Oct 25 '24 11:10 codecov[bot]

Run Java PreCommit

scwhittle avatar Oct 31 '24 14:10 scwhittle

Run Java_IOs_Direct PreCommit

scwhittle avatar Oct 31 '24 14:10 scwhittle

Run Java_Pulsar_IO_Direct PreCommit

scwhittle avatar Oct 31 '24 14:10 scwhittle

Run Java_IOs_Direct PreCommit

scwhittle avatar Nov 11 '24 16:11 scwhittle

Run Java_Hadoop_IO_Direct PreCommit

scwhittle avatar Nov 11 '24 16:11 scwhittle

Run Java PreCommit

scwhittle avatar Nov 11 '24 16:11 scwhittle

Run Java_Hadoop_IO_Direct PreCommit

scwhittle avatar Nov 11 '24 16:11 scwhittle

(removed, commented on wrong issue)

scwhittle avatar Nov 11 '24 21:11 scwhittle

Run Java PreCommit

scwhittle avatar Nov 11 '24 21:11 scwhittle

Run Java_IOs_Direct PreCommit

scwhittle avatar Nov 11 '24 21:11 scwhittle

Run Java_Pulsar_IO_Direct PreCommit

scwhittle avatar Nov 11 '24 21:11 scwhittle

Run Java PreCommit

scwhittle avatar Nov 12 '24 09:11 scwhittle

Run Java_IOs_Direct PreCommit

scwhittle avatar Nov 12 '24 09:11 scwhittle

precommit failures were unrelated flakes in :runners:google-cloud-dataflow-java:worker:test

scwhittle avatar Nov 12 '24 10:11 scwhittle