ozone icon indicating copy to clipboard operation
ozone copied to clipboard

HDDS-8784. trigger compaction outside of volume check.

Open guohao-rosicky opened this issue 9 months ago • 2 comments

What changes were proposed in this pull request?

Currently RocksDB compaction is triggered for schema v3 RocksDBs (one DB per volume) as part of the volume check. Periodic manual compaction is necessary because import of sst files as part of container replication can lead to many small sst files in RocksDB.

This operation may slow down the volume check and can have unintended consequences if the minimum gap between volume checks is set very high or very low. Ideally compaction should be triggered on its own independently controlled thread.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8784

How was this patch tested?

UT: org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer#testAutoCompactionSmallSstFile

guohao-rosicky avatar Apr 30 '24 10:04 guohao-rosicky

Thanks @guohao-rosicky for working on this. It looks like several tests are failing, please check.

https://github.com/guohao-rosicky/ozone/actions/runs/8892159377/job/24417356838#step:5:1252

https://github.com/guohao-rosicky/ozone/actions/runs/8892159377/job/24417354447#step:5:2662

https://github.com/guohao-rosicky/ozone/actions/runs/8892159377/job/24417354447#step:5:2875

https://github.com/guohao-rosicky/ozone/actions/runs/8892159377/job/24417357696#step:5:2144

adoroszlai avatar May 06 '24 11:05 adoroszlai

Change seems fine. Will wait for the metrics questions to get answered and then give it a try.

kerneltime avatar Jun 17 '24 21:06 kerneltime

I have already added the metric @kerneltime .

guohao-rosicky avatar Jul 25 '24 08:07 guohao-rosicky

LGTM + 1. Thanks @guohao-rosicky for the contribution, @kerneltime for the review.

ChenSammi avatar Aug 15 '24 03:08 ChenSammi