Graham Polley

Since porting to 2.1.0, Dataflow is leaving Datasets/Tables behind in BigQuery

9

Since porting to 2.1.0, Dataflow is leaving Datasets/Tables behind in BigQuery when the pipeline is cancelled or when it fails. We've been on 1.8.0/1.9.0 previous to this, and we've never...

Since porting to 2.1.0, pipeline now fails when it encounters an empty `PCollection` for BigQuery sink/output

1

We are porting some pipelines from `1.8.0` to `2.1.0` in our applications. These pipelines are basic: Read from GCS -> `ParDo` -> Write to BigQuery. There are multiple side outputs...

Pipeline is: BigQuery -> ParDo -> GCS Using SDK `2.1.0`. Inside the ParDo, calling `c.element().getF()` to get the `List` for the given `TableRow`, returns `null`: ``` public class BigQueryTableToOneFile {...

Corrections & suggestions

2

Here's some suggestions having worked hands-on for a few years with a decent amount of these tools. Also deleted a few duplicates (Cloud Build and Migrate for Anthos).

`DoFnTester` is deprecated Please fix