DataflowJavaSDK

DataflowJavaSDK copied to clipboard

Cannot stage dataflow job (java.net.SocketTimeoutException & java.io.IOException)

3

I am not able to run a Dataflow job with the DataflowRunner on GCP (I tried with the WordCount example and a custom dataflow of mine, without success). My configuration:...

mastronaute

Hi, I am using dataprep templates to invoke dataflow jobs using a cloud function with a GCS trigger (when new file arrive). There is constantly a strange behavior where the...

noamackerman

Backoff HTTP requests on IO Exceptions.

3

robertwb

From http://www.oracle.com/technetwork/java/javase/9-relnote-issues-3704069.html: > The application class loader is no longer an instance of > java.net.URLClassLoader (an implementation detail that was never > specified in previous releases). Code that assumes that...

cushon

Since porting to 2.1.0, pipeline now fails when it encounters an empty `PCollection` for BigQuery sink/output

1

We are porting some pipelines from `1.8.0` to `2.1.0` in our applications. These pipelines are basic: Read from GCS -> `ParDo` -> Write to BigQuery. There are multiple side outputs...

polleyg

Pipeline is: BigQuery -> ParDo -> GCS Using SDK `2.1.0`. Inside the ParDo, calling `c.element().getF()` to get the `List` for the given `TableRow`, returns `null`: ``` public class BigQueryTableToOneFile {...

polleyg

Reduce fn runner

1

killions-corp

Running `mvn package` on the starter archetype I see: ``` [INFO] Including org.hamcrest:hamcrest-core:jar:1.3 in the shaded jar. [WARNING] proto-google-common-protos-0.1.9.jar, proto-google-longrunning-v1-0.1.11.jar, grpc-google-common-protos-0.1.0.jar define 28 overlapping classes: [WARNING] - com.google.longrunning.ListOperationsRequestOrBuilder [WARNING] -...

elharo

This is an issue found in Apache Beam (https://issues.apache.org/jira/browse/BEAM-2708) and has been found to impact Dataflow SDK for Java 1.6.0 to 1.9.0. The fix has been backported with https://github.com/GoogleCloudPlatform/DataflowJavaSDK/pull/592

lukecwik

BigQueryIO.Write Setting table expiration time using Dataflow BigQuery sink

1

Setting table expiration time using Dataflow BigQuery sink, was not able to find anything in 1.9.0 for setting table expiration.

hassa687