spin
spin copied to clipboard
feat(oci): deduplicate layers prior to push; archive if needed
The following refactor-slash-feature brings the following:
(refactor)
- Remove extra 'layer_count' method for determining whether to employ archive layers when publishing. Instead, assemble layers assuming we won't need to use archive layers. After deduplicating the layers vec, if the count exceeds the max, then we re-assemble using archive layers. Since use of archive layers is the rarer case, it seemed fitting to only apply the tax of re-traversing the locked app here, instead of always doing so.
- Add
ClientOptsandcontent_ref_inline_max_sizeas first opt so that this value can be configured. Open to alternative ways to doing the same thing (admittedly, this need came about from unit testing).
(feature)
- Deduplicate assembled layers. For apps with multiple components and multiple static assets (think: static website), oftentimes the same component content or file content may be reused/included across multiple components. Previously, when publishing the app (via OCI), we would potentially be publishing the same content multiple times, which is both inefficient in bandwidth and final published artifact footprint. With the added deduplication, we ensure content is only published once and the OCI artifact is at its minimum size.
Depends on https://github.com/fermyon/oci-distribution/pull/1 (or alternative to achieve similar means)
TODO:
- [ ] Merge https://github.com/fermyon/oci-distribution/pull/1 and update reference in Cargo.toml (or explore alternative)
@vdice I added a "do-not-merge" label to flag the dependency on upstream OCI