ibis icon indicating copy to clipboard operation
ibis copied to clipboard

bug: array literal names are too long for bigquery

Open cpcloud opened this issue 1 year ago • 0 comments

What happened?

In [9]: from ibis.interactive import *

In [10]: con = ibis.connect("bigquery://ibis-gbq/workflows")

In [11]: con.execute(ibis.literal([1.0, 2.0, 3.0] * 100).unnest())
...
BadRequest: 400 Invalid field name "1_0_ 2_0_ 3_0_ 1_0_ 2_0_ 3_0_ 1_0_ 2_0_ 3_0_ 1_0_ 2_0_ 3_0_ 1_0_ 2_0_ 3_0_ 1_...". Fields must contain the allowed characters, and be at most 300 characters long. For allowed characters, please refer to https://cloud.google.com/bigquery/docs/schemas#column_names; reason: invalid, message: Invalid field name "1_0_ 2_0_ 3_0_ 1_0_ 2_0_ 3_0_ 1_0_ 2_0_ 3_0_ 1_0_ 2_0_ 3_0_ 1_0_ 2_0_ 3_0_ 1_...". Fields must contain the allowed characters, and be at most 300 characters long. For allowed characters, please refer to https://cloud.google.com/bigquery/docs/schemas#column_names

Ideally we can avoid this problem in general, but it should suffice to truncate literal value names up to some small fixed size and add a unique identifier on the end to allow disambiguation.

What version of ibis are you using?

main

What backend(s) are you using, if any?

BigQuery

Relevant log output

No response

Code of Conduct

  • [X] I agree to follow this project's Code of Conduct

cpcloud avatar Apr 11 '24 12:04 cpcloud