beam icon indicating copy to clipboard operation
beam copied to clipboard

Request to use BASIC enum when calling tables.get() in BigQuery #34075

Open tamannakakkar93 opened this issue 9 months ago • 6 comments

When calling BigQuery's tables.get method, today, we are retrieving FULL details (which is the default). However, this is expensive and is resulting in customers hitting Quota issues. BigQuery engineering team is open to relaxing this Quota if we retrieve using "BASIC" enum.

https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/get

beam/sdks/python/apache_beam/io/gcp/bigquery.py

Lines 902 to 903 in 9a1e7bf

table = bq.get_table( table_ref.projectId, table_ref.datasetId, table_ref.tableId) beam/sdks/python/apache_beam/io/gcp/bigquery_tools.py

Line 565 in 9a1e7bf

response = self.client.tables.Get(request)

tamannakakkar93 avatar Mar 11 '25 13:03 tamannakakkar93




self = <apitools.base.protorpclite.messages.EnumField object at 0x7945a7bebfa0> |  
  | value = 'BASIC' |  
  |   |  
  | def validate_element(self, value): |  
  | """Validate single element of field. |  
  |   |  
  | This is different from validate in that it is used on individual |  
  | values of repeated fields. |  
  |   |  
  | Args: |  
  | value: Value to validate. |  
  |   |  
  | Returns: |  
  | The value casted in the expected type. |  
  |   |  
  | Raises: |  
  | ValidationError if value is not expected type. |  
  | """ |  
  | if not isinstance(value, self.type): |  
  |   |  
  | # Authorize int values as float. |  
  | if isinstance(value, six.integer_types) and self.type == float: |  
  | return float(value) |  
  |   |  
  | if value is None: |  
  | if self.required: |  
  | raise ValidationError('Required field is missing') |  
  | else: |  
  | try: |  
  | name = self.name |  
  | except AttributeError: |  
  | raise ValidationError('Expected type %s for %s, ' |  
  | 'found %s (type %s)' % |  
  | (self.type, self.__class__.__name__, |  
  | value, type(value))) |  
  | else: |  
  | >                   raise ValidationError( |  
  | 'Expected type %s for field %s, found %s (type %s)' % |  
  | (self.type, name, value, type(value))) |  
  | E                   apitools.base.protorpclite.messages.ValidationError: Expected type <class 'apache_beam.io.gcp.internal.clients.bigquery.bigquery_v2_messages.BigqueryTablesGetRequest.ViewValueValuesEnum'> for field view, found BASIC (type <class 'str'>) |  
  |  

<br class="Apple-interchange-newline">

Try to use apache_beam.io.gcp.internal.clients.bigquery.bigquery_v2_messages.BigqueryTablesGetRequest.ViewValueValuesEnum not the string.

liferoad avatar Apr 11 '25 17:04 liferoad

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Aug 03 '25 12:08 github-actions[bot]

keep open

tamannakakkar93 avatar Aug 05 '25 07:08 tamannakakkar93

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Oct 05 '25 12:10 github-actions[bot]

keep open

On Sun, Oct 5, 2025 at 6:16 PM github-actions[bot] @.***> wrote:

github-actions[bot] left a comment (apache/beam#34249) https://github.com/apache/beam/pull/34249#issuecomment-3369034585

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the @.*** list. Thank you for your contributions.

— Reply to this email directly, view it on GitHub https://github.com/apache/beam/pull/34249#issuecomment-3369034585, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMJ65DJLJJPJBFMCKUNCGID3WEHKFAVCNFSM6AAAAABYY4QH7SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNRZGAZTINJYGU . You are receiving this because you authored the thread.Message ID: @.***>

--

Tamanna Kakkar Technical Solutions Engineer-Big Data

tamannakakkar93 avatar Oct 07 '25 15:10 tamannakakkar93

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Dec 08 '25 12:12 github-actions[bot]

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

github-actions[bot] avatar Dec 15 '25 12:12 github-actions[bot]