apache
[Feature] Introduce New Metrics for Compaction
Search before asking
- [X] I searched in the issues and found nothing similar.
Motivation
During recent usage, we have observed that DelVector's operations are dependent on compaction, making it a critical factor for data visibility. However, the current metrics related to compaction are still limited. To address this, I propose introducing the following new metrics to enrich our compaction-related metrics:
Standard Deviation of Compaction Time: Measures the variability in compaction times to help identify any extreme cases (e.g., some tasks running particularly slow).
Ratio of Compacted Input to Output Files: Helps to understand the effectiveness of compaction.
Change Rate in Total File Size Before and After Compaction: Assesses the impact of compaction on disk space, especially when compressing many small files into fewer large files.
Memory Usage: Measures memory usage (such as heap memory) at the start and end of a compaction task to calculate changes in memory usage during the compaction process.[It can be monitored through the k8s process.]
Number of Pending Compaction Tasks: Monitors the length of the task queue submitted to the executor to understand if there are task congestion issues.
Compaction Failure Rate: Records the proportion of compaction tasks that encounter errors, which is important for monitoring the stability and reliability of compaction tasks.
CPU and I/O Utilization: Combines system monitoring tools to record CPU and I/O utilization during the execution of compaction tasks. This helps to identify the impact of compaction tasks on system resources.[It can be monitored through the k8s process.]
Average Compaction Time: Calculates the average execution time of compaction tasks over a period.
Minimum Compaction Time: Records the shortest compaction time observed.
Maximum Compaction Time: Records the longest compaction time observed.
I believe these metrics will provide us with a more comprehensive understanding of compaction performance and its impact on our system.
Solution
No response
Anything else?
No response
Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
