[C++] Support GCP BigQuery Storage API

[asfimport]

Docs: https://cloud.google.com/bigquery/docs/reference/storage/ 

Need to investigate the best way to do this maybe just see if we can build our client on GCP (once a protobuf definition is published to https://github.com/googleapis/googleapis/tree/master/google)?)

 

This will serve as a parent issue, and sub-issues will be added for subtasks if necessary.

Reporter: Micah Kornfield / @emkornfield

Related issues:

• [C++] Implement converter between Arrow record batches and Avro records (is blocked by)

_{Note: This issue was originally created as ARROW-4668. Please see the migration documentation for further details.}

[asfimport]

Micah Kornfield / @emkornfield: It looks like protos are available in the python library: https://github.com/googleapis/google-cloud-python/blob/master/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage.proto

[asfimport]

Antoine Pitrou / @pitrou: Is this different from the GCS API referred to in ARROW-1231 ?

[asfimport]

Wes McKinney / @wesm: Yeah, it's different. GCS is generic blob storage (like Amazon S3). BigQuery is a SQL-based data warehouse product. Its internal storage is not exposed to customers as files, so you have to use the storage API to access the raw data without necessarily running SQL queries

[asfimport]

Micah Kornfield / @emkornfield: Wes is correct.  I'll also add that either this (or even a higher level wrapper around BQ) or flight would make a good test case for DataSet APIs to make sure they are generic enough.  I won't be getting to this anytime soon, so I'm going to unassign it from myself.  I have some sample code on my work computer that I will also try to share to show how the API can be accessed in a simple scenario.

[github-actions[bot]]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.