beam icon indicating copy to clipboard operation
beam copied to clipboard

[Feature Request]: Emit warnings when pcollection elements approach or exceed 2GB limits.

Open tvalentyn opened this issue 1 year ago • 0 comments

What would you like to happen?

We added the warnings in Python SDK: https://github.com/apache/beam/issues/31607#issuecomment-2169043225 but to my knowledge, we don't have them in Java or Go, which might result in confusing errors when individual elements exceed 2GB, which exceeds protobuf limits.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • [ ] Component: Python SDK
  • [X] Component: Java SDK
  • [X] Component: Go SDK
  • [X] Component: Typescript SDK
  • [ ] Component: IO connector
  • [ ] Component: Beam YAML
  • [ ] Component: Beam examples
  • [ ] Component: Beam playground
  • [ ] Component: Beam katas
  • [ ] Component: Website
  • [ ] Component: Infrastructure
  • [ ] Component: Spark Runner
  • [ ] Component: Flink Runner
  • [ ] Component: Samza Runner
  • [ ] Component: Twister2 Runner
  • [ ] Component: Hazelcast Jet Runner
  • [ ] Component: Google Cloud Dataflow Runner

tvalentyn avatar Aug 27 '24 18:08 tvalentyn