amoro
amoro copied to clipboard
[Improvement][Optimize]: build file metrics
Search before asking
- [X] I have searched in the issues and found no similar issues.
What would you like to be improved?
Optimize plan have to load all files from table file_info_cache frequently, which may bring performance problems when there are many files in file cache.
How should we improve?
Add some files metrics to avoid load all files from table file_info_cache, which help to decide whether it's necessary to optimize before load all files from file_info_cache.
These metrics(including small files cnt in each partition...) can be generated by listener in optimize service, which listen to the changes of file cache service.
Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct