Radosław Stankiewicz

Goal - Bring more flexibility/strategies into incremental models. updatePartitionFilter is not enough. Additional insert-overwrite strategy with DML (delete) before model execution. Delete statement should select partitions dynamically or based on...

Mongo to BigQuery - set schema manually in case js udf is filtering data

1

MongoDB to JDBC has an issue with schema detection when UDF is used for filtering - #1328 . Good workaround is providing schema directly, similarly to JDBC, Oracle to BigQuery.

PubsubMessageWithTopicCoder.of() is returning wrong coder

6

PubsubMessageWithTopicCoder should return PubsubMessageWithTopicCoder PubsubMessageWithAttributesAndMessageIdCoder. While investigating Dynamic Destinations on Direct runner I found out that PubsubMessageWithTopicCoder is never used and topic is lost and pipeline fails. fixes #31679

[Bug]: PubsubMessageWithTopicCoder.of() returns PubsubMessageWithAttributesAndMessageIdCoder

1

### What happened? PubsubMessageWithTopicCoder should return PubsubMessageWithTopicCoder while it returns PubsubMessageWithAttributesAndMessageIdCoder While investigating Dynamic Destinations on Direct runner I found out that PubsubMessageWithTopicCoder is never used and topic is lost...

Mongo to BQ has poor support for filters - they are on dataflow side with help of UDFs. This makes it expensive to run for Mongo Database and is expensive...

Concat protos in BQStorageWriteAPI - solve edge cases during mering of nested repeated fields

17

BQStorageWriteAPI concatenates proto payload with unknown fields to support schema update. today concat is naive and won't work correctly for repeated structs. This change introduces expensive merging only when Beam...

[Bug]: JMSIO.Write should better handle connection issues

2

### What happened? Currently JMS.Write catches all JMSException and routes the input records to failedMessagesTags. Issues: - there is no way to understand why record has failed, input record doesn't...

[OpenTelemetry] Model changes to allow OpenTelemtry context propagation

4

Changes: - dependencies definitions to otel - WindowedValue interface change to expose context for propagation - context is not read by user - changes are behind external_metadata experiment Next PRs:...

[Task]: Propagate context from windmill to windowedValue

1

### What needs to happen? Propagate context from windmill to windowedValue https://github.com/apache/beam/blob/81bb5066bab883f79328e52f0d2a55e9b90f2f65/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedWindmillReader.java#L145 ### Issue Priority Priority: 2 (default / most normal work should be filed as P2) ### Issue Components...

[Drain] TimerData causedByDrain field

4

This is PR is adding new field to TimerData. Noop for most of implementation apart from runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillTimerInternals.java that propagates draining information to TimerData. ------------------------ Thank you for your contribution! Follow...