amoro icon indicating copy to clipboard operation
amoro copied to clipboard

[AMORO-2540] Report expiring snapshots event for Iceberg and mixed format

Open xxubai opened this issue 1 year ago • 3 comments

Why are the changes needed?

Close #2540 .

Brief change log

  • Report expire snapshots event to downstream listeners

How was this patch tested?

  • [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • [ ] Add screenshots for manual tests if appropriate

  • [x] Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not documented)

xxubai avatar Feb 05 '24 09:02 xxubai

Codecov Report

Attention: Patch coverage is 74.33628% with 58 lines in your changes are missing coverage. Please review.

Project coverage is 32.61%. Comparing base (2808413) to head (f15c295).

Files Patch % Lines
...etease/arctic/server/events/MixedExpireEvents.java 73.68% 16 Missing and 4 partials :warning:
...s/api/events/mixed/ExpireMixedSnapshotsResult.java 0.00% 7 Missing :warning:
.../optimizing/maintainer/IcebergTableMaintainer.java 89.70% 4 Missing and 3 partials :warning:
.../netease/arctic/server/utils/IcebergTableUtil.java 57.14% 6 Missing :warning:
...netease/arctic/ams/api/events/ExpireOperation.java 0.00% 5 Missing :warning:
.../ams/api/events/iceberg/ExpireSnapshotsResult.java 0.00% 5 Missing :warning:
...ms/api/events/iceberg/RemoveOrphanFilesResult.java 0.00% 3 Missing :warning:
...a/com/netease/arctic/ams/api/events/EventType.java 0.00% 2 Missing :warning:
...ease/arctic/ams/api/events/IcebergReportEvent.java 0.00% 2 Missing :warning:
...com/netease/arctic/ams/api/events/ExpireEvent.java 0.00% 1 Missing :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2541      +/-   ##
============================================
+ Coverage     32.44%   32.61%   +0.16%     
- Complexity     4412     4433      +21     
============================================
  Files           590      596       +6     
  Lines         50055    50260     +205     
  Branches       6622     6640      +18     
============================================
+ Hits          16242    16391     +149     
- Misses        32533    32578      +45     
- Partials       1280     1291      +11     
Flag Coverage Δ
core 30.71% <74.33%> (+0.18%) :arrow_up:
trino 50.93% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Feb 06 '24 07:02 codecov[bot]

Hi, thanks for this wonderful feature.

I wonder if you can somehow explain the motivation of this definition?

majin1102 avatar Mar 28 '24 06:03 majin1102

I wonder if you can somehow explain the motivation of this definition?

The purpose is to report information about files that have been deleted after the expiration timestamp, such as the path to the file, so that downstream systems can monitor or utilize this information for appropriate operations. But after discussing with @zhoujinsong , I think the reported information should be as concise as possible, and also need to wait for the API to be refactored to handle it.

xxubai avatar Mar 28 '24 08:03 xxubai

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Aug 22 '24 00:08 github-actions[bot]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

github-actions[bot] avatar Aug 30 '24 00:08 github-actions[bot]