ibis
ibis copied to clipboard
bug: BigQuery backend table.to_pyarrow_batches is not honoring the chunk_size parameter
What happened?
for following code for bigquery:
c = ibis.bigquery.connect(...)
t = c.table('some table')
for y in t.to_pyarrow_batches(chunk_size=1_000_000): # <- change this parameter does not take effect
print(y.num_rows)
for our case, the record returned is always 4k rows instead of larger set we expect. tried same code for Duckdb/Snowflake, both can return proper sized pyarrow table
What version of ibis are you using?
9.0.0
What backend(s) are you using, if any?
BigQuery
Relevant log output
No response
Code of Conduct
- [X] I agree to follow this project's Code of Conduct