iotdb icon indicating copy to clipboard operation
iotdb copied to clipboard

[IOTDB-3164] Manage the memory of cross space compaction in write memory controller

Open THUMarkLau opened this issue 2 years ago • 3 comments

See IOTDB-3164.

This PR is one part of the compaction memory management. It does the following jobs:

  • Allocate memory before cross space compaction, and release the memory after the execution.
  • Remove the RewriteCompactionFileSelector, which is deduplicated with RewriteCrossSpaceCompactionSelector.

To complete the compaction memory management, we need to complete the following jobs:

  • Manage the memory usage of unsequence inner space compaction, which uses read point performer.
  • Design and implement the memory estimator for read chunk performer.
  • Manage the memory usage of sequence inner space comapction, which uses read chunk performer.

THUMarkLau avatar Aug 08 '22 09:08 THUMarkLau

Coverage Status

Coverage increased (+0.02%) to 42.338% when pulling ad73693498b616abc847d1e8988464175c373ee7 on IOTDB-3164-O into 37d6cfed32b4aa2858c898dcb012b824f3ab041e on master.

coveralls avatar Aug 08 '22 12:08 coveralls

I use IoTDB benchmark tests changed after the performance, and the master (commit: a8f757d6e4fecc2251d2042a025960bf29c6e0f3) are compared. The result of the comparison is shown in the figure below. In terms of average performance, the write performance after the change decreases by about 2.8%.

master branch e173a42189cf91925f9f73475afac2b

with memory management for cross space compaction dd9c26571d2659edfb92e40e6d87df9

THUMarkLau avatar Aug 10 '22 05:08 THUMarkLau

https://apache-iotdb.feishu.cn/docx/doxcnuh5hy6ee9Qb7uLCYXkFDdc

qiaojialin avatar Aug 26 '22 15:08 qiaojialin