scio icon indicating copy to clipboard operation
scio copied to clipboard

Support clustering in saveAs..BigQueryTable

Open dmateusp opened this issue 2 years ago • 0 comments

I am trying to write to a clustered table using saveAsTypedBigQueryTable and getting:

<table> is configured to use TableDestinationCoderV2. Set withClustering() on BigQueryIO.write() and,  if you provided a custom DynamicDestinations instance, override getDestinationCoder() to return TableDestinationCoderV3.

I only hit this when I am processing many partitions at once. I see that the SCIO API doesn't expose the withClustering option.

https://beam.apache.org/releases/javadoc/2.31.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withClustering-com.google.api.services.bigquery.model.Clustering-

dmateusp avatar Jul 27 '21 07:07 dmateusp