beam
beam copied to clipboard
[Bug]: UnsupportedOperationException when spin up beam-examples-multi-language expansion service
What happened?
To reproduce, run
wget https://repo1.maven.org/maven2/org/apache/beam/beam-examples-multi-language/2.59.0/beam-examples-multi-language-2.59.0.jar
java -jar beam-examples-multi-language-2.59.0.jar 33333
there are folowing "INFO" level log with exception:
INFO: Could not load the TransformPayloadTranslator org.apache.beam.runners.dataflow.DataflowRunner$DataflowPayloadTranslator@32f0fba8 to the Expansion Service.
java.lang.UnsupportedOperationException: URN of DataflowPayloadTranslator depends on the transform. Please use 'getUrn(PTransform transform)' instead.
at org.apache.beam.runners.dataflow.DataflowRunner$DataflowPayloadTranslator.getUrn(DataflowRunner.java:2644)
at org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader.knownTransforms(ExpansionService.java:173)
at org.apache.beam.sdk.expansion.service.ExpansionService.loadRegisteredTransforms(ExpansionService.java:574)
at org.apache.beam.sdk.expansion.service.ExpansionService.getRegisteredTransforms(ExpansionService.java:560)
at org.apache.beam.sdk.expansion.service.ExpansionService.main(ExpansionService.java:834)
Found during 2.60.0 validation. Looks like the exceptions are ignored and does not affect the functionality of the expansion service. There is still log indicating the cross-lang transforms are registered:
INFO: Registering external transforms: [beam:transform:org.apache.beam:spanner_insert_or_update:v1, beam:directrunner:transforms:test_stream:v1, beam:transform:org.apache.beam:pubsublite_write:v1, beam:transform:org.apache.beam:kafka_write:v1, beam:transform:combine_grouped_values:v1, beam:transform:combine_globally:v1, beam:external:java:generate_sequence:v1, beam:transform:redistribute_by_key:v1, beam:directrunner:transforms:gabw:v1, beam:transform:window_into:v1, beam:transform:org.apache.beam:spanner_update:v1, beam:transform:org.apache.beam:pubsub_write:v1, beam:schematransform:org.apache.beam:kafka_read:v1, beam:schematransform:org.apache.beam:kafka_write:v1, beam:transform:combine_per_key:v1, beam:transform:org.apache.beam:kafka_read_with_metadata:v1, beam:transform:group_by_key:v1, beam:transform:pubsub_read:v1, beam:transform:org.apache.beam:spanner_delete:v1, beam:transform:group_into_batches:v1, beam:transform:org.apache.beam:spanner_read:v1, beam:transform:pubsub_write:v1, beam:transform:org.apache.beam:spanner_insert:v1, beam:transform:group_into_batches_with_sharded_key:v1, beam:transform:pubsub_write:v2, beam:transform:create_view:v1, beam:transform:teststream:v1, beam:transform:sdf_process_keyed_elements:v1, beam:transform:org.apache.beam:javacount:v1, beam:transform:org.apache.beam:pubsub_read:v1, beam:transform:org.apache.beam:javaprefix:v1, beam:directrunner:transforms:write_view:v1, beam:transform:org.apache.beam:spanner_replace:v1, beam:directrunner:transforms:gbko:v1, beam:transform:org.apache.beam:pubsublite_read:v1, beam:transform:org.apache.beam:bigquery_write:v1, beam:transform:flatten:v1, beam:transform:impulse:v1, beam:runners_core:transforms:splittable_process:v1, beam:transform:write_files:v1, beam:directrunner:transforms:stateful_pardo:v1, beam:transform:pardo:v1, beam:transform:org.apache.beam:bigquery_read:v1, beam:transform:org.apache.beam:kafka_read_without_metadata:v1, beam:transform:reshuffle:v1, beam:transform:redistribute_arbitrarily:v1]
...
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner