beam icon indicating copy to clipboard operation
beam copied to clipboard

[Feature Request]: Support int32 for properties of Cross lang transformations written in Java

Open Naireen opened this issue 8 months ago • 0 comments

What would you like to happen?

All cross lang transformations that are implemented in java that have a python wrapper that use the old ExternalTransform API such as KafkaIO are forced to have int64 properties since the translation layer converts a python int to int64. So if an int32 property is added to the java transform, unless the python wrapper specifies an np.int32 (which is clunky since it needlessly adds an np dependency), the python pipeline will error.

Issue comes from this file: https://github.com/apache/beam/blob/0f2e1963987f1fbb3329016d8c862639ed4fbe43/sdks/python/apache_beam/typehints/schemas.py#L4

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

  • [X] Component: Python SDK
  • [X] Component: Java SDK
  • [ ] Component: Go SDK
  • [ ] Component: Typescript SDK
  • [ ] Component: IO connector
  • [ ] Component: Beam YAML
  • [ ] Component: Beam examples
  • [ ] Component: Beam playground
  • [ ] Component: Beam katas
  • [ ] Component: Website
  • [ ] Component: Spark Runner
  • [ ] Component: Flink Runner
  • [ ] Component: Samza Runner
  • [ ] Component: Twister2 Runner
  • [ ] Component: Hazelcast Jet Runner
  • [ ] Component: Google Cloud Dataflow Runner

Naireen avatar Jun 24 '24 14:06 Naireen