dbt-clickhouse
dbt-clickhouse copied to clipboard
Flaky test test_clickhouse_errors.py::TestOOMError::test_oom, Unexpected EOF while reading bytes
Describe the bug
Test fails with
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes'
+ where 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes' = RunResult(status=<RunStatus.Error: 'error'>, timing=[TimingInfo(name='compile', started_at=datetime.datetime(2025, 12,...ey=[], time_spine=None, previous_batch_results=None, batch=None, _has_this=None), agate_table=None, batch_results=None).message
I can see them happening with the Native/TCP (port 9440) connection. I have not been able to reproduce it with the regular HTTP (port 8443) connection
Looks like this may be related to some of the existing EOF situations that are also found in other languaje clients, but I have not dig enough on it.
Examples
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/20085306976/job/57621291453
- https://github.com/ClickHouse/dbt-clickhouse/actions/runs/19844815183/job/56860067429
Steps to reproduce
Use the Native protocol (DBT_CH_TEST_PORT=9440) and run
pytest tests/integration/adapter/clickhouse/test_clickhouse_errors.py --flake-finder --flake-runs=1000 -n=3
In the terminal you can see different errors:
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[121] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[129] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[89] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[201] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[137] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[65] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[104] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[176] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[74] - AssertionError: assert 'exceeded' in 'Runtime Error in model oom_table (models/oom_table.sql)\n ClickHouse exception: Unexpected EOF while reading bytes'
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[34] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[114] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[75] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[202] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[226] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[105] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
FAILED tests/integration/adapter/clickhouse/test_clickhouse_errors.py::TestOOMError::test_oom[35] - dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Searching for bad queries, I see the ones that the test creates:
select *
from system.query_log
WHERE
event_date = today()
and event_time > now() - INTERVAL 1 hour
and type > 2
and current_database like '%dbt_clickhouse%'
That returns as expected
Code: 241. DB::Exception: (total) memory limit exceeded: would use 32.99 GiB (attempt to allocate chunk of 32.00 GiB bytes), current RSS: 1011.44 MiB, maximum: 14.40 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker: while executing 'FUNCTION repeat(toString(__table1.number) :: 0, 100000_UInt32 :: 1) -> repeat(toString(__table1.number), 100000_UInt32) String : 2'. (MEMORY_LIMIT_EXCEEDED) (version 25.8.1.8702 (official build))
ClickHouse server
- ClickHouse Server version: 25.8.1.8702