Ahmed Abualsaud

Add Iceberg workflows

1

Adding workflows for Iceberg integration and load tests. Integration test already exists. Load test will be added in #31392

Add iceberg load test

6

Adding a load test for IcebergIO Separating integration test (from #31220) into its own suite

Default SchemaTransform configs to snake_case

4

Bringing back the changes reverted in #31109 More details in #31061 and #31353

Kafka SchemaTransform translation

4

Adding SchemaTransform translation and tests for KafkaIO

[Task]: Support more Beam portable schema types as Python types

3

### What needs to happen? Beam portable schemas include primitive and more complex types (represented as [logical types](https://github.com/apache/beam/tree/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes)). Some of these types are supported in the Python SDK: https://github.com/apache/beam/blob/99202b237e364bf77f40b6da0ec22cb7b17c37d0/sdks/python/apache_beam/typehints/schemas.py#L23-L41 When...

Pass-through IcebergIO catalog properties

1

The current IcebergIO implementation supports only a fixed set of catalog properties. This is an unnecessary limitation for users and in some cases makes the connector unusable (e.g. if an...

Add options to control number of Storage API connections when using multiplexing

3

Controls min and max number of connections to connection pool. For more details on this BigQuery Storage Write API feature, see https://cloud.google.com/bigquery/docs/write-api-best-practices#connection_pool_management

Support writing to Pubsub with ordering key; Add PubsubMessage SchemaCoder

40

Fixes #21162 I wasn't able to use the existing `PubsubMessageWithAttributesAndMessageIdCoder` because it doesn't encode/decode the message's topic, which is needed for dynamic destinations. There are already a number of existing...

[Python] Managed Transforms API

4

Adding Python API for Managed transforms, similar to the [Java API](https://github.com/ahmedabu98/beam/blob/bb4ad994304d0bb5f31196f3bc2e4e3616ae59bb/sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java#L76).