golang-samples
golang-samples copied to clipboard
GoogleCloudPlatform
bigquery: demonstrate use of arrow serialization with bigquerystorage reading
Copied from https://github.com/googleapis/google-cloud-go/issues/3682, filed by https://github.com/mrgleeco.
Request:
A golang analog to the Java implementation as described at https://cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1
What we need:
Example showing how to use https://pkg.go.dev/github.com/apache/arrow/go/arrow and relevant subpackages for reading individual row batches, and at least some schema handling code.
https://arrow.apache.org/docs/status.html#implementation-status tracks the deltas by language. Looks like we're still missing some necessary types for go (null and decima256), and it looks like there's still a requirement that the flatbuffer to be decoded has the schema embedded (the storage API uses an external schema message). Need to either work around this via constructing/merging schema into a flatbuffer or go fix arrow upstream.
@shollyman looks like implementation for Decimal256 was recently added to Go ( see https://github.com/apache/arrow/commit/6d1bc624454b8a83e5c90410b51c2718059c394b ) and the Null data type also appears on the implementation status dashboard. I'll take a look later on the schema part that you mentioned here.
Resolved by https://github.com/GoogleCloudPlatform/golang-samples/pull/2710