beam

beam copied to clipboard

### What happened? It is found that https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml has been timing out since Oct 30, but wasn't detected by the tool (it should reopen #34990) ~relevant code path:~ ~https://github.com/apache/beam/blob/8ff2c940321cc8adb49e2fc590a1d35fd7113add/.test-infra/tools/flaky_test_detection.py#L83~ EDIT:...

Abacn

[Bug]: BigQueryIO performance overhead on DynamicDestinations on parsing json

7

### What happened? Where: https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinationsHelpers.java#L305-L314 Why: validation of static valueprovider is per element Impact: 20% overhead on this stage. Request: store outcome of validation within the object itself. ### Issue...

stankiewicz

GCS client library migration in Java SDK - part 1

4

This PR lays the groundwork for the GCS client library migration in the Java SDK. It refactors the existing GCS client implementation to prepare for a new implementation while maintaining...

shunping

[Feature Request]: Add yaml sql example using calcite_connection_properties, and document this in the transform catalog

2

### What would you like to happen? Right now, any SQL options have to be passed through as pipeline options. This is somewhat awkward for yaml/xlang - for example: ```...

damccorm

### What happened? context: #36995 Currently, the spanner client version is pinned at 6.104.0 for fixing an issue in the version from the current libraries-bom (#35868). Once we have a...

shunping

test: update libraries-bom

2

Beam is already on 4.33(https://github.com/apache/beam/blob/3ce2abdc505d09569b668d8071f89306ea9aeb81/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L641). Running `python3 scripts/tools/bomupgrader.py 26.72.0` to update libraries-bom ``` INFO:root:-----Update BeamModulePlugin----- INFO:root:Unchanged: com.google.api:gax:2.72.1 INFO:root:Changed com.google.cloud:google-cloud-spanner: 6.95.1 -> 6.103.0 INFO:root:Unchanged: io.grpc:grpc-netty:1.76.0 INFO:root:Unchanged: io.netty:netty-transport:4.1.124.Final INFO:root:Unchanged: io.opentelemetry:opentelemetry-sdk:1.52.0 INFO:root:Unchanged: com.google.protobuf:protobuf-java:4.33.0...

mpeddada1

Update ExternalJavaProvider.available method to support Windows systems

1

The current implementation of the [ExternalJavaProvider.available](https://github.com/apache/beam/blob/365c2d92965c5e23c23d6e1f3c7a1cd048c872d8/sdks/python/apache_beam/yaml/yaml_provider.py#L323) method only supports Unix/Linux systems. This issue aims to update the function to also support Windows systems. **Changes**: The available method will be updated...

eug

Add environment variable for skipping slow ensure_clean_state python fixture

4

The `ensure_clean_state` pytest fixture adds ~200ms of overhead to every unit test. This slows down local development quite a bit. This adds an opt-in environment flag to skip this fixture...

hjtran

Fix ProtoCoder NoSuchMethodException

7

**Please** add a meaningful description for your change here ------------------------ Handles `NoSuchMethodException` in `ProtoCoder.getParser()` when `getDefaultInstance()` is unavailable by adding a fallback to instantiation. Addresses https://github.com/apache/beam/issues/19713 Thank you for your...

Suvrat1629

[Dataflow Streaming] Move timer tag logic to WindmillStateTagUtil

3

This is a preparation to introduce a new version of WindmillStateTagUtil. No logic changes, just moving code around.

arunpandianp