beam icon indicating copy to clipboard operation
beam copied to clipboard
verified publisher icon apache

[Feature Request]: Reduce FnApi error logs for processing that is cancelled for expected reasons

Open scwhittle opened this issue 8 months ago • 5 comments

What would you like to happen?

The runner of a portable beam pipeline may start processing a bundle that it later wants to cancel. Possible reasons for this could be:

  • hedging
  • active work is invalidated due to runner restart/scaling and needs to retry

This cancellation is currently activated by returning an error on a state or data request for the bundle. This ends up cancelling the processing of the bundle but can result in logs as there is no way to distinguish such runner initiated cancellations from real errors.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • [x] Component: Python SDK
  • [x] Component: Java SDK
  • [ ] Component: Go SDK
  • [ ] Component: Typescript SDK
  • [ ] Component: IO connector
  • [ ] Component: Beam YAML
  • [ ] Component: Beam examples
  • [ ] Component: Beam playground
  • [ ] Component: Beam katas
  • [ ] Component: Website
  • [ ] Component: Infrastructure
  • [ ] Component: Spark Runner
  • [ ] Component: Flink Runner
  • [ ] Component: Samza Runner
  • [ ] Component: Twister2 Runner
  • [ ] Component: Hazelcast Jet Runner
  • [x] Component: Google Cloud Dataflow Runner

scwhittle avatar Apr 22 '25 11:04 scwhittle

.take-issue

scwhittle avatar Apr 22 '25 11:04 scwhittle

From discussion on beam-dev it sounds like there is agreement that adding a ErrorReason to the state request response instead of just error string would be useful for improving logging and error handling within sdks.

scwhittle avatar Jun 10 '25 11:06 scwhittle

This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.

github-actions[bot] avatar Nov 07 '25 12:11 github-actions[bot]

This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.

github-actions[bot] avatar Dec 08 '25 12:12 github-actions[bot]

I still plan on doing this once I get a chance

scwhittle avatar Dec 08 '25 14:12 scwhittle