Ahmed Abualsaud

Pass user specified destination type to UpdateSchemaDestination

3

Fixes #22543 When using multiple partitions for a BigQuery `FILE_LOADS` write, the user's dynamic destination type isn't properly passed into `UpdateSchemaDestination`. This leads to a `ClassCastException` when attempting to get...

Allow non-repeated JSON fields for BigQuery FILE_LOADS

8

JSON insertion via batch loads has until recently only been supported via CSV format. It is now supported with JSON and Avro format (see [here for documentation](https://cloud.google.com/bigquery/docs/reference/standard-sql/json-data#use_a_batch_load_job)). It does not...

Fix failing BQ file loads test for Py37

9

Fixes #23264

Generate external transform wrappers using a script

22

Implementing a script that generates wrappers for external SchemaTransforms, according to Option #3 in the following design doc: https://s.apache.org/autogen-wrappers The script's workflow takes place in setup.py, which can be invoked...

[Bug]: MongoDB to BigQuery batch template only considers the first document when generating the BQ write schema

12

### Related Template(s) MongoDbToBigQuery ### What happened? When using this template, the implicit assumption is that all documents in the MongoDB collection have the same schema. The template takes the...

[Bugfix] Replace only the last occurrence of Datetime patterns in a directory path

3

While using a template that utilizes DLQ, found some cases where dead-letter GCS directory names get altered unexpectedly if they contain Datetime patterns. for example: DLQ would take `gs://bucket/dead-letter/recommendation` and...

[Failing Test]: Python Precommit failing due to RRIO test failures

1

### What happened? Python PreCommit [1] and Coverage [2] test suites are failing with the following error: ``` ERROR apache_beam.io.requestresponse_it_test:requestresponse_it_test.py:304 Unable to start redis container for RRIO tests. ``` [1]...

[Improvement]: (Python BQ batch loads) Perform one copy job for a set of temp tables instead of one copy job per temp table

3

### What would you like to happen? For large Java BQ batch loads that require copy jobs, the temp tables created for a given destination are [grouped up into a...