Use native RAFS in `nydusify commit` instead of tarfs and custom tarball logic

[markasoftware-tc]

Overview

This PR replaces much of the nydusify commit logic by using existing features in nydus-image. This simplifies the code and improves performance.

Related Issues

No related issue.

Change Details

The existing nydusify commit logic uses Go to stream a tarball of modified files and whiteouts into nydus-image create in tarfs mode. Without historical context I don't know exactly why this was done, but my guess is that nydus-image didn't handle overlayfs-format whiteouts at the time nydusify commit was written. This logic duplicates some of the work nydus-image is about to do anyway, and isn't implemented in the most efficient way (for example, it stats all upper-layer files on the lower layer as well, to check if they are identical).

This PR just calls nydus-image create --whiteout-spec overlayfs to do effectively the same thing. In my benchmarks, this typically improves performance by at least 25% on fast disks.

nydus-image create has the --parent-bootstrap option performs a merge internally, so this PR also removes the explicit calls to nydus-image merge to simplify things further.

Test Results

I have manually tested committing, including with deleted files, for correctness.

Change Type

Please select the type of change your pull request relates to:

• [ ] Bug Fix
• [ ] Feature Addition
• [ ] Documentation Update
• [X] Code Refactoring
• [X] Performance Improvement
• [ ] Other (please describe)

Self-Checklist

Before submitting a pull request, please ensure you have completed the following:

• [ ] I have run a code style check and addressed any warnings/errors.
• [ ] I have added appropriate comments to my code (if applicable).
• [ ] I have updated the documentation (if applicable).
• [ ] I have written appropriate unit tests.

Remaining items I'm unsure about:

• Haven't done super extensive testing in general.
• Removed the WithoutFiles option because there's no equivalent on nydus-image create. If this is an important feature, then we might be out of luck (unless we add the feature to nydus-image create)

[imeoer]

Hi @markasoftware-tc, thanks for the PR, we initially considered using the diff approach for the lower and upper dirs, is the fn Changes in contrib/nydusify/pkg/committer/diff/overlay_linux.go, because it implements stricter and complete overlayfs key features, such as checkRedirect and checkOpaque. But the nydus-image create --whiteout-spec overlayfs only consider about the upper dir, it's not a complete implementation for overlayfs diff. :(

[codecov[bot]]

Codecov Report

:x: Patch coverage is 0% with 239 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 55.40%. Comparing base (6e49ad0) to head (d83e296). :warning: Report is 18 commits behind head on master.

Files with missing lines	Patch %	Lines
contrib/nydusify/pkg/committer/commiter.go	0.00%	232 Missing :warning:
contrib/nydusify/cmd/nydusify.go	0.00%	7 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1776      +/-   ##
==========================================
+ Coverage   54.86%   55.40%   +0.54%     
==========================================
  Files         199      195       -4     
  Lines       54089    53555     -534     
  Branches    44705    44705              
==========================================
- Hits        29675    29673       -2     
+ Misses      22929    22396     -533     
- Partials     1485     1486       +1     

Files with missing lines	Coverage Δ
contrib/nydusify/cmd/nydusify.go	15.77% <0.00%> (+0.01%)	:arrow_up:
contrib/nydusify/pkg/committer/commiter.go	0.00% <0.00%> (ø)

... and 2 files with indirect coverage changes

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions[bot]]

This PR is stale because it has been open 60 days with no activity.

[github-actions[bot]]

This PR was closed because it has been stalled for 7 days with no activity.