spin icon indicating copy to clipboard operation
spin copied to clipboard

feat(oci): deduplicate layers prior to push; archive if needed

Open vdice opened this issue 1 year ago • 1 comments

The following refactor-slash-feature brings the following:

(refactor)

  • Remove extra 'layer_count' method for determining whether to employ archive layers when publishing. Instead, assemble layers assuming we won't need to use archive layers. After deduplicating the layers vec, if the count exceeds the max, then we re-assemble using archive layers. Since use of archive layers is the rarer case, it seemed fitting to only apply the tax of re-traversing the locked app here, instead of always doing so.
  • Add ClientOpts and content_ref_inline_max_size as first opt so that this value can be configured. Open to alternative ways to doing the same thing (admittedly, this need came about from unit testing).

(feature)

  • Deduplicate assembled layers. For apps with multiple components and multiple static assets (think: static website), oftentimes the same component content or file content may be reused/included across multiple components. Previously, when publishing the app (via OCI), we would potentially be publishing the same content multiple times, which is both inefficient in bandwidth and final published artifact footprint. With the added deduplication, we ensure content is only published once and the OCI artifact is at its minimum size.

Depends on https://github.com/fermyon/oci-distribution/pull/1 (or alternative to achieve similar means)

TODO:

  • [ ] Merge https://github.com/fermyon/oci-distribution/pull/1 and update reference in Cargo.toml (or explore alternative)

vdice avatar Mar 25 '24 20:03 vdice

@vdice I added a "do-not-merge" label to flag the dependency on upstream OCI

itowlson avatar Mar 26 '24 19:03 itowlson