beam

beam copied to clipboard

[Python] Managed Transforms API

4

Adding Python API for Managed transforms, similar to the [Java API](https://github.com/ahmedabu98/beam/blob/bb4ad994304d0bb5f31196f3bc2e4e3616ae59bb/sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java#L76).

ahmedabu98

There are folks out there trying to run Beam Python on Databricks [1]. While there is documentation out there for the Java SDK [2], Python is more involved because the...

damccorm

PubsubMessageWithTopicCoder.of() is returning wrong coder

6

PubsubMessageWithTopicCoder should return PubsubMessageWithTopicCoder PubsubMessageWithAttributesAndMessageIdCoder. While investigating Dynamic Destinations on Direct runner I found out that PubsubMessageWithTopicCoder is never used and topic is lost and pipeline fails. fixes #31679

stankiewicz

[Bug]: Using string in PartitionColumn throws error, tries to convert it to string

4

### What happened? Referencing [JdbcIo's doc](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jdbc/JdbcIO.html) in the section for Parallel reading from a JDBC datasource, It mentions to use either of these types of column for paritioning `Beam supports...

ssaurav-redhat

Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.13.8 to 1.17.2 in /sdks

1

Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.13.8 to 1.17.2. Changelog Sourced from github.com/aws/aws-sdk-go-v2/feature/s3/manager's changelog. Release (2022-12-02) General Highlights Dependency Update: Updated to the latest SDK module versions Module Highlights github.com/aws/aws-sdk-go-v2/service/appsync: v1.17.0 Feature: Fixes...

dependabot[bot]

Added insertion and enrichment pipeline

3

WIP: Added insertion and enrichment pipeline for RAG (Retrival Augmented Generation) usecase #GSOC-259 ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and...

itsayushpandey

Basic yaml-defined provider.

3

This allows one to define new transforms with YAML that can be imported and used in other pipelines. ------------------------ Thank you for your contribution! Follow this checklist to help us...

robertwb

[Dataflow Streaming] Enabled Heartbeat by Default

3

This sets heartbeat work by default and will rollout along next beam release. This CL also refactors and fixed some unit test issues.

TongruiLi

Support withFormatRecordOnFailureFunction() for BigQuery STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE methods

3

Fixes #31354 ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate issue in your description...

Amar3tto

[Feature Request]: UDF supports lambda & RecordType parameter

1

### What would you like to happen? I want to create a new UDF which supports a lambda parameter and a Row parameter, for example ```java static class LambdaUDFA implements...

gabry-lab