goccy
failed to add query result to dynamic destination table (on a simple query)
What happened?
I have a table, pretty big, over 4M rows (if it's important the table contains arrays with simple types, like REPEATED/INTEGER).
So, I am making a query:
bq --api http://0.0.0.0:9050 query --project_id=company "SELECT count(*) FROM team.Items"
(other queries are affected too, for example:
{"query": "SELECT avg(orderedQuantity) FROM team.Items", "values": []}
and getting response:
BigQuery error in query operation: failed to add query result to dynamic destination table: dataset bqjob_r5323c5b3273d9d3f_0000018e1d743054_1 is already created
Logs on the emulator are:
2024-03-08T10:45:37.660+0100 INFO contentdata/repository.go:167 {"query": "SELECT count(*) FROM team.Items", "values": []}
2024-03-08T10:45:58.684+0100 DEBUG contentdata/repository.go:246 query result {"rows": [[4449208]]}
2024-03-08T10:45:58.705+0100 INFO server/middleware.go:63 POST /projects/company/jobs {"query": "alt=json"}
2024-03-08T10:45:58.715+0100 INFO contentdata/repository.go:167 {"query": "SELECT count(*) FROM team.Items", "values": []}
2024-03-08T10:46:13.453+0100 DEBUG contentdata/repository.go:246 query result {"rows": [[4449208]]}
2024-03-08T10:46:13.453+0100 ERROR server/handler.go:1018 jobInternalError {"error": "jobInternalError: failed to add query result to dynamic destination table: dataset bqjob_r5323c5b3273d9d3f_0000018e1d743054_1 is already created"}
What did you expect to happen?
I'm expecting the result to be returned.
How can we reproduce it (as minimally and precisely as possible)?
- Prepare table with over 4M rows
- Execute query (python API / bq client)
- Error occures every time
Anything else we need to know?
No response
It seems to me like the BigQuery client is retrying your query as the server closed the connection. Judging by the timestamps in your logs, I'm guessing you're running into this issue: #213 / #266 @goccy Do you have any opinions on #266?
BQ client allows specifying the job timeout (--job_timeout_ms=100000), which is ignored by the emulator.
The python lib has this possibility as well:
def query(
....
timeout: TimeoutType = DEFAULT_TIMEOUT
)
---
DEFAULT_RETRY = retry.Retry(predicate=_should_retry, deadline=600.0)
"""The default retry object.
Any method with a ``retry`` parameter will be retried automatically,
with reasonable defaults. To disable retry, pass ``retry=None``.
To modify the default retry behavior, call a ``with_XXX`` method
on ``DEFAULT_RETRY``. For example, to change the deadline to 30 seconds,
pass ``retry=bigquery.DEFAULT_RETRY.with_deadline(30)``.
"""
DEFAULT_TIMEOUT = None
"""The default API timeout.
This is the time to wait per request. To adjust the total wait time, set a
deadline on the retry object.
"""