professional-services icon indicating copy to clipboard operation
professional-services copied to clipboard
verified publisher icon GoogleCloudPlatform

Implement Dataflow CsvIO example with Custom Template

Open damondouglas opened this issue 3 years ago • 1 comments

This PR provides two demonstrations. The first is an example processing CSV files into Schema aware PCollections using Apache Beam where we expect different headers. Please consult examples/dataflow-csvio/README for an explanation of this problem and what this example solves.

The second is an example how to build and deploy the example as a custom Dataflow template using gradle in the context of terraform. The community has expressed that documented Beam Java SDK examples only use maven. Using Infrastructure-as-Code with terraform allows one to integrate the building of custom Dataflow templates within the workflow where one provisions dependent and related Google Cloud resources.

damondouglas avatar Jul 19 '22 23:07 damondouglas

@prathapreddy123 Thank you again for reviewing. I've updated per most of the comments and this is ready for your review.

damondouglas avatar Aug 04 '22 21:08 damondouglas

This has been in the backlog for too long. I will submit to the Beam project instead. Thank you, again.

damondouglas avatar Sep 22 '22 22:09 damondouglas