[Feature Request]: Request to use BASIC enum when calling tables.get() in BigQuery

[PraveenAkunuru]

What would you like to happen?

When calling BigQuery's tables.get method, today, we are retrieving FULL details (which is the default). However, this is expensive and is resulting in customers hitting Quota issues. BigQuery engineering team is open to relaxing this Quota if we retrieve using "BASIC" enum.

https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/get

https://github.com/apache/beam/blob/9a1e7bf4cd399177e5de7fe28bad741cade452a5/sdks/python/apache_beam/io/gcp/bigquery.py#L902-L903

https://github.com/apache/beam/blob/9a1e7bf4cd399177e5de7fe28bad741cade452a5/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L565

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• [x] Component: Python SDK
• [ ] Component: Java SDK
• [ ] Component: Go SDK
• [ ] Component: Typescript SDK
• [x] Component: IO connector
• [ ] Component: Beam YAML
• [ ] Component: Beam examples
• [ ] Component: Beam playground
• [ ] Component: Beam katas
• [ ] Component: Website
• [ ] Component: Infrastructure
• [ ] Component: Spark Runner
• [ ] Component: Flink Runner
• [ ] Component: Samza Runner
• [ ] Component: Twister2 Runner
• [ ] Component: Hazelcast Jet Runner
• [ ] Component: Google Cloud Dataflow Runner

[tamannakakkar93]

.take-issue

[github-actions[bot]]

This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.

[liferoad]

keep this open

[tamannakakkar93]

working on it

[BalajiReddy1]

Hi, I investigated this issue and found a blocker.

It appears that the generated client library in the current codebase is outdated. Specifically, the BigqueryTablesGetRequest object in apache_beam.io.gcp.internal.clients.bigquery does not currently have the view field defined.

The client library (bigquery_v2_messages.py) needs to be regenerated to include the BASIC view enum before this code change can be implemented. I am unable to run the generation scripts in my current environment, so I am leaving this for someone with the appropriate setup to handle.