amoro icon indicating copy to clipboard operation
amoro copied to clipboard

[AMORO-2273]: Fix highly memory usage when orphan file clean is enabled with too many manifest list file

Open baiyangtx opened this issue 2 years ago • 4 comments

Why are the changes needed?

Close #2273.

Brief change log

  • Remove in-memory Content file cache for mixed-format Table Maintainer.
  • Refactor logic of fetch all manifest file info from a snapshot when doing orphan file clean.

How was this patch tested?

  • [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • [ ] Add screenshots for manual tests if appropriate

  • [X] Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (no)

baiyangtx avatar Nov 21 '23 11:11 baiyangtx

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Nov 22 '23 08:11 CLAassistant

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (38c4986) 53.46% compared to head (798ca0a) 53.53%. Report is 1 commits behind head on master.

Files Patch % Lines
.../netease/arctic/server/utils/IcebergTableUtil.java 90.00% 2 Missing :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2346      +/-   ##
============================================
+ Coverage     53.46%   53.53%   +0.07%     
- Complexity     4358     4371      +13     
============================================
  Files           517      517              
  Lines         29861    29870       +9     
  Branches       2908     2909       +1     
============================================
+ Hits          15964    15992      +28     
+ Misses        12608    12590      -18     
+ Partials       1289     1288       -1     
Flag Coverage Δ
core 54.02% <93.93%> (+0.09%) :arrow_up:
trino 50.93% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Nov 23 '23 08:11 codecov[bot]

Thanks @baiyangtx for contributions, I left some comment,BTW the ci is unpass

shidayang avatar Nov 29 '23 07:11 shidayang

CI unpass

shidayang avatar Dec 04 '23 02:12 shidayang