ydb icon indicating copy to clipboard operation
ydb copied to clipboard

Memory pages leak in dq_output_consumer

Open Hor911 opened this issue 1 year ago • 1 comments

  • After 1st run of TPC-H Q18 scale 1000 each node allocated ~8GB
  • Each next run adds 2-3GB more
  • RM's Quota is successfully reset no 0
  • A lot of runs causes OOM as result
  • Problem is fixed with change of GetYqlMemoryPool to arrow default memory pool here:

https://github.com/ydb-platform/ydb/blob/1ba20dfbc7e53fde7ea4ba67ee49065fab557f65/ydb/library/yql/dq/runtime/dq_output_consumer.cpp#L541

Hor911 avatar Aug 29 '24 06:08 Hor911

Looks like memory pages in YQL Pool are not reused under some conditions

For investigation tracing of alloc/free in this point is needed

Hor911 avatar Aug 29 '24 06:08 Hor911

periodical pools cleanup was added. memory control is reworking under different issues.

gridnevvvit avatar Nov 28 '24 10:11 gridnevvvit