fix(trino): add delay time to avoid Trino issue
I found the Trino container usually query failed due to nodes is empty.
I tried to execute SELECT * FROM system.runtime.nodes and SHOW CATALOGS LIKE 'tpch' before querying to ensure that the node was active and the 'tpch' catalog was ready. It still encountered the problem. This issue may be related to a Trino discovery node problem. A similar issue was reported previously (see https://github.com/trinodb/trino/discussions/13388)
The final solution is sleeping for a few seconds.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Please upload report for BASE (
main@82a2e7b). Learn more about missing BASE report.
Additional details and impacted files
@@ Coverage Diff @@
## main #735 +/- ##
=======================================
Coverage ? 85.58%
=======================================
Files ? 12
Lines ? 666
Branches ? 104
=======================================
Hits ? 570
Misses ? 74
Partials ? 22
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
🚨 Try these New Features:
- Flaky Tests Detection - Detect and resolve failed and flaky tests
what exception does it throw? can't that exception be added to wait_container_is_ready or one of the other decorator we have and it will catch that and rerun the waiter method? this module seems to be abusing the decorators a bit here....
The raised exception is when we use the Trino client to connect the container. We can't catch it.
conn = connect(
host=db.get_container_host_ip(),
port=db.get_exposed_port(db.port),
user="test",
)
cur = conn.cursor()
> cur.execute("CREATE TABLE memory.default.orders AS SELECT * from tpch.tiny.orders")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../.venv/lib/python3.11/site-packages/trino/dbapi.py:589: in execute
self._iterator = iter(self._query.execute())
../.venv/lib/python3.11/site-packages/trino/client.py:829: in execute
self._result.rows += self.fetch()
../.venv/lib/python3.11/site-packages/trino/client.py:849: in fetch
status = self._request.process(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <trino.client.TrinoRequest object at 0x115a2c210>
http_response = <Response [200]>
def process(self, http_response) -> TrinoStatus:
if not http_response.ok:
self.raise_response_error(http_response)
http_response.encoding = "utf-8"
response = http_response.json()
if "error" in response:
> raise self._process_error(response["error"], response.get("id"))
E trino.exceptions.TrinoQueryError: TrinoQueryError(type=INTERNAL_ERROR, name=GENERIC_INTERNAL_ERROR, message="nodes is empty", query_id=20241118_033536_00001_tf8kg)
../.venv/lib/python3.11/site-packages/trino/client.py:621: TrinoQueryError
The trino logs in the container
2024-11-18 11:46:00 2024-11-18T03:46:00.216Z INFO main io.trino.server.Server Server startup completed in 7.66s
2024-11-18 11:46:00 2024-11-18T03:46:00.216Z INFO main io.trino.server.Server ======== SERVER STARTED ========
2024-11-18 11:46:00 2024-11-18T03:46:00.844Z INFO dispatcher-query-3 io.trino.event.QueryMonitor TIMELINE: Query 20241118_034600_00000_g6qa4 :: FINISHED :: elapsed 379ms :: planning 119ms :: waiting 26ms :: scheduling 216ms :: running 32ms :: finishing 12ms :: begin 2024-11-18T03:46:00.425Z :: end 2024-11-18T03:46:00.804Z
2024-11-18 11:46:01 2024-11-18T03:46:01.070Z INFO dispatcher-query-5 io.trino.event.QueryMonitor TIMELINE: Query 20241118_034600_00001_g6qa4 :: FAILED (GENERIC_INTERNAL_ERROR) :: elapsed 197ms :: planning 197ms :: waiting 0ms :: scheduling 0ms :: running 0ms :: finishing 0ms :: begin 2024-11-18T03:46:00.871Z :: end 2024-11-18T03:46:01.068Z
You can see the SERVER STARTED log; the first query is FINISHED but the second is FAILED.
But the first query is SELECT 1. Maybe I can use tpch.tiny.nation and wait for it. WDYT?
deadline = time.time() + c.max_tries
while time.time() < deadline:
try:
cur = conn.cursor()
cur.execute("SELECT * FROM tpch.tiny.nation LIMIT 1")
cur.fetchall()
return
except Exception:
time.sleep(c.sleep_time)
raise TimeoutError(f"Trino did not start within {c.max_tries:.3f} seconds")
Hi @alexanderankin, could you recheck this PR?