beam icon indicating copy to clipboard operation
beam copied to clipboard

[Bug]: UnsupportedOperationException when spin up beam-examples-multi-language expansion service

Open Abacn opened this issue 1 year ago • 0 comments

What happened?

To reproduce, run

wget https://repo1.maven.org/maven2/org/apache/beam/beam-examples-multi-language/2.59.0/beam-examples-multi-language-2.59.0.jar

java -jar beam-examples-multi-language-2.59.0.jar 33333

there are folowing "INFO" level log with exception:

INFO: Could not load the TransformPayloadTranslator org.apache.beam.runners.dataflow.DataflowRunner$DataflowPayloadTranslator@32f0fba8 to the Expansion Service.
java.lang.UnsupportedOperationException: URN of DataflowPayloadTranslator depends on the transform. Please use 'getUrn(PTransform transform)' instead.
	at org.apache.beam.runners.dataflow.DataflowRunner$DataflowPayloadTranslator.getUrn(DataflowRunner.java:2644)
	at org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader.knownTransforms(ExpansionService.java:173)
	at org.apache.beam.sdk.expansion.service.ExpansionService.loadRegisteredTransforms(ExpansionService.java:574)
	at org.apache.beam.sdk.expansion.service.ExpansionService.getRegisteredTransforms(ExpansionService.java:560)
	at org.apache.beam.sdk.expansion.service.ExpansionService.main(ExpansionService.java:834)

Found during 2.60.0 validation. Looks like the exceptions are ignored and does not affect the functionality of the expansion service. There is still log indicating the cross-lang transforms are registered:

INFO: Registering external transforms: [beam:transform:org.apache.beam:spanner_insert_or_update:v1, beam:directrunner:transforms:test_stream:v1, beam:transform:org.apache.beam:pubsublite_write:v1, beam:transform:org.apache.beam:kafka_write:v1, beam:transform:combine_grouped_values:v1, beam:transform:combine_globally:v1, beam:external:java:generate_sequence:v1, beam:transform:redistribute_by_key:v1, beam:directrunner:transforms:gabw:v1, beam:transform:window_into:v1, beam:transform:org.apache.beam:spanner_update:v1, beam:transform:org.apache.beam:pubsub_write:v1, beam:schematransform:org.apache.beam:kafka_read:v1, beam:schematransform:org.apache.beam:kafka_write:v1, beam:transform:combine_per_key:v1, beam:transform:org.apache.beam:kafka_read_with_metadata:v1, beam:transform:group_by_key:v1, beam:transform:pubsub_read:v1, beam:transform:org.apache.beam:spanner_delete:v1, beam:transform:group_into_batches:v1, beam:transform:org.apache.beam:spanner_read:v1, beam:transform:pubsub_write:v1, beam:transform:org.apache.beam:spanner_insert:v1, beam:transform:group_into_batches_with_sharded_key:v1, beam:transform:pubsub_write:v2, beam:transform:create_view:v1, beam:transform:teststream:v1, beam:transform:sdf_process_keyed_elements:v1, beam:transform:org.apache.beam:javacount:v1, beam:transform:org.apache.beam:pubsub_read:v1, beam:transform:org.apache.beam:javaprefix:v1, beam:directrunner:transforms:write_view:v1, beam:transform:org.apache.beam:spanner_replace:v1, beam:directrunner:transforms:gbko:v1, beam:transform:org.apache.beam:pubsublite_read:v1, beam:transform:org.apache.beam:bigquery_write:v1, beam:transform:flatten:v1, beam:transform:impulse:v1, beam:runners_core:transforms:splittable_process:v1, beam:transform:write_files:v1, beam:directrunner:transforms:stateful_pardo:v1, beam:transform:pardo:v1, beam:transform:org.apache.beam:bigquery_read:v1, beam:transform:org.apache.beam:kafka_read_without_metadata:v1, beam:transform:reshuffle:v1, beam:transform:redistribute_arbitrarily:v1]

...

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • [ ] Component: Python SDK
  • [X] Component: Java SDK
  • [ ] Component: Go SDK
  • [ ] Component: Typescript SDK
  • [ ] Component: IO connector
  • [ ] Component: Beam YAML
  • [ ] Component: Beam examples
  • [ ] Component: Beam playground
  • [ ] Component: Beam katas
  • [ ] Component: Website
  • [ ] Component: Infrastructure
  • [ ] Component: Spark Runner
  • [ ] Component: Flink Runner
  • [ ] Component: Samza Runner
  • [ ] Component: Twister2 Runner
  • [ ] Component: Hazelcast Jet Runner
  • [ ] Component: Google Cloud Dataflow Runner

Abacn avatar Oct 15 '24 18:10 Abacn