datalens icon indicating copy to clipboard operation
datalens copied to clipboard

Memory limit (total) exceeded in Dataset preview

Open handgunman opened this issue 2 months ago • 0 comments

When adding computed fields with aggregation to the dataset preview, a query is built that consumes excessive memory in Clickhouse. A query of the form: default query: SELECT sum(t1.sumfield) AS res_0, ... t1.id AS res_4, t1.datecommit AS res_5, t1.ndssum AS res_6, t1.numberfield AS res_7 ... FROM "default".cheks_v AS t1 GROUP BY res_1, res_2, res_3, res_4, res_5, res_6, res_7, ...

In fact, grouping is performed on all fields of the sample. And only then the limitation is done. If the table size is comparable to the RAM size, an error occurs. Since grouping by all fields is performed, you should change the query to this structure:

select sum(a), a,b,c from (select a,b,c from t limit 10) group by a,b,c

limit before group by

Otherwise, it is almost impossible to use the preview on large amounts of data.

handgunman avatar May 03 '24 07:05 handgunman