qryn icon indicating copy to clipboard operation
qryn copied to clipboard

Clickhouse Overcommit error

Open tsearle opened this issue 5 months ago • 10 comments

Qryn threw the following error Code: 241. DB::Exception: Memory limit (total) exceeded: would use 7.21 GiB (attempt to allocate chunk of 4476253 bytes), maximum: 7.20 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: (while reading column labels): (while reading from part data/3bebf7bf-6abc-4a92-ae16-834c90cbafbc/20240925_0_24268_13230/ in table qryn.time_series (3bebf7bf-6abc-4a92-ae16-834c90cbafbc) located on disk s3WithKeeperDisk of type s3, from mark 0 with max_rows_to_read = 8192): While executing MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread). (MEMORY_LIMIT_EXCEEDED) (version 24.5.1.22957 (official build))

When digging on the clickhouse console, the query that failed appears to be... WITH idx AS (select fingerprint from qryn.time_series_gin as time_series_gin where ((((key = 'name') and (val = 'kamailio_app_server_dead'))) and (date >= toDate(fromUnixTimestamp(1727294240))) and (date <= toDate(fromUnixTimestamp(1727294540))) and (type in (0,0))) group by fingerprint having (groupBitOr(bitShiftLeft(((key = 'name') and (val = 'kamailio_app_server_dead'))::UInt64, 0)) = 1)), raw AS (select argMaxMerge(last) as value,fingerprint,intDiv(timestamp_ns, 15000000000) * 15000 as timestamp_ms from metrics_15s as metrics_15s where ((fingerprint in (idx)) and (timestamp_ns >= 1727294240000000000) and (timestamp_ns <= 1727294540000000000) and (type in (0,0))) group by fingerprint,timestamp_ms order by fingerprint,timestamp_ms), timeSeries AS (select fingerprint,arraySort(JSONExtractKeysAndValues(labels, 'String')) as labels from qryn.time_series where ((fingerprint in (idx)) and (type in (0,0)))) select any(labels) as stream,arraySort(groupArray((raw.timestamp_ms, raw.value))) as values from raw as raw any left join timeSeries as time_series on time_series.fingerprint = raw.fingerprint group by raw.fingerprint order by raw.fingerprint FORMAT RowBinary

This caused the prometheus query (for alarm monitoring) to fail: kamailio_app_server_dead{}

Am I doing something stupidly wrong?

tsearle avatar Sep 26 '24 06:09 tsearle