DataflowJavaSDK

DataflowJavaSDK copied to clipboard

Take 2: Error when reading from External Unbounded Source and Write to PubSubIO

5

I took the title from a previous issue #105 that has been resolved and it sounds just like my issue. I have a pipeline that reads from unbonded pubsubio and...

cbazza

Dataflow discarding massive amount of events due to Window object or inner processing

5

Been recently developing a Dataflow consumer which read from a PubSub subscription and outputs to Parquet files the combination of all those objects grouped within the same window. While I...

turboT4

Google Cloud Dataflow removing accents and special chars with '??'

1

This is going to be quite a hit or miss question as I don't really know which context or piece of code to give you as it is a situation...

turboT4

Could you please revert the deletion of source code? This code is still in production and the repo should exist and be public. Once that's no longer true it should...

elharo

Dataflow uses incorrect full file size with GS file using Content-Encoding: gzip

6

To reproduce: - Upload a simple file (10000 sequential numbers, one per line) to Google storage specifying GZIP compression `gsutil cp -Z numbers.txt gs:///numbers.txt`. - Execute a simple dataflow just...

rfevang

BigQueryIO.write fails when destination has partition decorator

1

Following is the code that writes to BigQuery: ``` BigQueryIO.writeTableRows() .to(destination) .withCreateDisposition(CREATE_IF_NEEDED) .withWriteDisposition(WRITE_APPEND) .withSchema(tableSchema) .expand(tableRows); ``` Here's the destination's implementation: ``` public TableDestination apply(ValueInSingleWindow input) { String partition = timestampExtractor.apply(input.getValue())...

darshanmehta10

PubsubMessage readed form PubsubIO have no attributes

1

Dataflow SDK version: 2.4.0 I haven't tested with other attribute but when we use these attribute for de-duplication and timestaming we got null PubsubMessage's attribute For example ``` PubsubIO.readMessages() .fromSubscription("mine")...

chikien276

Google API Client Library version 1.23.0 causes runtime problems with Dataflow Java SDK

19

The new Google API Client Library, version 1.23.0, appears to cause problems with the Dataflow Java SDK when submitting and/or running jobs. This appears to affect Dataflow Java SDKs in...

moandcompany

GCS temp location to store temp files

15

trying to run BigQueryTornadoes and keep having this error : [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project google-cloud-dataflow-java-examples-all: An exception occured while executing the Java clas s. null:...

demousteven

Dataflow /var/opt/google/dataflow directory created as /var/opt/google/dataflow/dataflow

1

Dataflow is not aboutable to find files packaged with my classes. I use Class.getResource("/data.json"). Stackdriver log shows it's looking for the file in /var/opt/google/dataflow/some-random-jar-name.jar!/data.json. When I ssh into the VM...

junying1