Graham Polley comments

Results 7 comments of


                                            Graham Polley

Since porting to 2.1.0, Dataflow is leaving Datasets/Tables behind in BigQuery

Only seen it in batch so far, and cannot reproduce yet.

Since porting to 2.1.0, Dataflow is leaving Datasets/Tables behind in BigQuery

I was just thinking about this today because it happened yet again. Agree, auto expire on the datasets makes sense.

Dataflow jobs using the SDK for Java 1.6.0 and reading compressed files from TextIO with compression mode set may be subject to data loss.

Hi Dan. How can we identify which pipelines, if any, have lost data? We have many pipelines reading GZIP files from GCS. Graham

Dataflow jobs using the SDK for Java 1.6.0 and reading compressed files from TextIO with compression mode set may be subject to data loss.

Thanks Dan. As recommended, we're going to roll out a hotfix with the compression type set to `AUTO` for now.

Dataflow jobs using the SDK for Java 1.6.0 and reading compressed files from TextIO with compression mode set may be subject to data loss.

Awesome stuff Dan. Great turnaround time.

Google API Client Library version 1.23.0 causes runtime problems with Dataflow Java SDK

We've had the same problem. Except for us, it was with the the BigQuery API that we were bringing into our project. Removing it fixed it (Beam has a dependancy...

Corrections & suggestions

My pleasure. Glad I could help. Love this cheat sheet. I share it all the time with our customers and new GCP engineers. -gp On Tue, 14 Apr 2020 at...