beam icon indicating copy to clipboard operation
beam copied to clipboard

Support runners other than DirectRunner for :examples:java:execute

Open Abacn opened this issue 6 months ago • 8 comments

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • [ ] Update CHANGES.md with noteworthy changes.
  • [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels Python tests Java tests Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Abacn avatar Jun 12 '25 20:06 Abacn

Tested with

./gradlew :examples:java:execute -PmainClass=org.apache.beam.examples.WordCount -Pexec.args=" \
       --runner=DataflowRunner \
       --region=us-central1 \
       --tempLocation=gs://.../tmp/ \
       --project=... \
       --jobName=$USER-wordcount-job
       --output=gs://.../output/output.txt \
       --experiments=use_staged_dataflow_worker_jar

Abacn avatar Jun 12 '25 20:06 Abacn

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

github-actions[bot] avatar Jun 12 '25 21:06 github-actions[bot]

R: @shunping

Java PreCommit currently failing due to #35274 #35233

Abacn avatar Jun 13 '25 14:06 Abacn

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

github-actions[bot] avatar Jun 13 '25 14:06 github-actions[bot]

Looks like the last successful run is three days ago.

https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java.yml?query=branch%3Amaster+is%3Asuccess

shunping avatar Jun 13 '25 15:06 shunping

Trying to revert the first red master run (https://github.com/apache/beam/actions/runs/15566528898). Let's see if the check can pass with this. https://github.com/apache/beam/pull/35276

shunping avatar Jun 13 '25 15:06 shunping

It was due to #35233 and #35252, the latter one is fixed. Random PrismRunner tests still failing

Abacn avatar Jun 13 '25 18:06 Abacn

* What went wrong:
Execution failed for task ':runners:prism:java:test'.
> There were failing tests. See the report at: file:///runner/_work/beam/beam/runners/prism/java/build/reports/tests/test/index.html

BUILD FAILED in 13m 16s

windowing (org.apache.beam.runners.prism.PrismRunnerTest) failed

runners/prism/java/build/test-results/test/TEST-org.apache.beam.runners.prism.PrismRunnerTest.xml [took 2s]

org.apache.beam.vendor.grpc.v1p69p0.io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at app//org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
	at app//org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
	at app//org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
	at app//org.apache.beam.model.jobmanagement.v1.JobServiceGrpc$JobServiceBlockingStub.run(JobServiceGrpc.java:717)

latest run there is a single prism runner test failing, not related to this change

PTAL @shunping

Abacn avatar Jun 13 '25 19:06 Abacn

Given the previous failed Java PreCommit tests on prism have been fixed at https://github.com/apache/beam/pull/35336, I think it is safe to LGTM here.

shunping avatar Jun 17 '25 19:06 shunping

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 54.50%. Comparing base (de76cfa) to head (8b6756a). Report is 59 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##             master   #35262   +/-   ##
=========================================
  Coverage     54.50%   54.50%           
  Complexity     1479     1479           
=========================================
  Files          1017     1017           
  Lines        160983   160983           
  Branches       1079     1079           
=========================================
  Hits          87737    87737           
  Misses        71140    71140           
  Partials       2106     2106           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Jun 17 '25 20:06 codecov[bot]