ydb
ydb copied to clipboard
Memory pages leak in dq_output_consumer
- After 1st run of TPC-H Q18 scale 1000 each node allocated ~8GB
- Each next run adds 2-3GB more
- RM's Quota is successfully reset no 0
- A lot of runs causes OOM as result
- Problem is fixed with change of GetYqlMemoryPool to arrow default memory pool here:
https://github.com/ydb-platform/ydb/blob/1ba20dfbc7e53fde7ea4ba67ee49065fab557f65/ydb/library/yql/dq/runtime/dq_output_consumer.cpp#L541
Looks like memory pages in YQL Pool are not reused under some conditions
For investigation tracing of alloc/free in this point is needed
periodical pools cleanup was added. memory control is reworking under different issues.