feat(warehouse)!: artifact discovery and manual Freight creation

TL;DR

This pull request addresses the majority of the backend part of #761, by reflecting the last 20 discovered artifacts for each subscription of a Warehouse in its Status, allowing the UI to present the user with options to pick from to "mix-and-match" their selection of artifacts to create Freight for.

Combined with these changes, a "manual" and "automatic" Freight creation mode has been introduced to the Warehouse, in addition to changes to the reconciliation model, and further optimizations to the logic that gathers information about upstream artifacts.

API changes

.status.discoveredArtifacts

For each subscription of a Warehouse, Kargo will discover the 20 latest artifacts based on the selection strategy and reflect them in .status.discoveredArtifacts field of the object. The UI can use this information to provide the user with the option to "mix and match" artifacts for which to create Freight.

An example:

discoveredArtifacts:
  charts:
    - repoURL: oci://ghcr.io/akuity/kargo-charts/kargo
      versions:
        - 0.6.0
        - 0.6.0-rc.2
        - 0.6.0-rc.1
        # ...omitted for brevity
  commits:
    - repoURL: https://github.com/akuity/kargo.git
      commits:
        - author: Remington Breeze <[email protected]>
          branch: main
          createdAt: "2024-05-15T18:58:00Z"
          id: c5c74d42069b41461763bc3c21de02b579ada1dc
          subject: 'fix(ui): Promotion confirmation message overflowed freight box (#2015)'
        - author: Remington Breeze <[email protected]>
          branch: main
          createdAt: "2024-05-15T18:55:00Z"
          id: f65a27fcb5bc48861d10768313bb5e4201a8cb86
          subject: 'feat(ui): Ability to abort verification from UI (#2013)'
        # ...omitted for brevity
  images:
    - repoURL: ghcr.io/akuity/kargo
      images:
        - digest: sha256:eae672841a1756683a967b7c25eab31d694ac66ecd09b35191664ec166530f1a
          tag: v0.6.0
        - digest: sha256:79103eb0596ac4c6e12f2eedd8e9066d8557ff86ba58b9a4037d5bdbd74c1ac0
          tag: v0.6.0-rc.2
        - digest: sha256:0ee72b9e8955a7bf4f4b78667ebe9612983e3d0d184eab14f75b1b87c3de31d9
          tag: v0.6.0-rc.1
        # ...omitted for brevity

Note: Not all fields for images and commits are shown above, as the availability of data (e.g. a Git tag, or the creation timestamp of a container image) depends on the chosen selection strategy. Please refer to the in-code changes for further details on all fields.

.spec.freightCreation

This newly introduced field configures if Freight is automatically created for newly discovered artifacts. It can be set to Automatic (the default) to discover artifacts and automatically create Freight for the latest discovered version(s), or to Manual to discover artifacts without creating Freight.

Behavioral changes

• Warehouse reconciler will first discover the latest artifacts for each subscription, and then (optionally) automatically create Freight for it.
• The discovery of Git artifacts required having access to tree or repository history in many cases. Because of this, and to continue to keep things performant, the cloning process has changed from using a "shallow clone" (--depth=1) to the newer "partial clone" (--filter) option. To read more about this, please refer to this excellent article.
• Git metadata of tags and commits is now gathered using git for-each-ref and git log with specifically designed custom formatters. Instead of performing FS-heavy git checkout, etc. operations.
• Instead of looking at the diff for all commits between the current Freight and HEAD to determine if it contains changes to included and/or excluded paths. Every individual commit is inspected until 20 matching commits are found (or no commits remain).

Other notable changes

• For commits, we do now take note of the author.

May 09 '24 16:05 hiddeco

✅ Deploy Preview for docs-kargo-akuity-io ready!

Name	Link
🔨 Latest commit	e44c41ab23ca06c44ccaee6ad1923484684727d5
🔍 Latest deploy log	https://app.netlify.com/sites/docs-kargo-akuity-io/deploys/664614c7933e8500083fa79f
😎 Deploy Preview	https://deploy-preview-1984.kargo.akuity.io
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

May 09 '24 16:05 netlify[bot]

Codecov Report

Attention: Patch coverage is 54.62478% with 260 lines in your changes are missing coverage. Please review.

Project coverage is 45.80%. Comparing base (1e3c3e7) to head (e44c41a). Report is 25 commits behind head on main.

@@            Coverage Diff             @@
##             main    #1984      +/-   ##
==========================================
- Coverage   46.21%   45.80%   -0.42%     
==========================================
  Files         235      235              
  Lines       16017    16191     +174     
==========================================
+ Hits         7403     7417      +14     
- Misses       8254     8413     +159     
- Partials      360      361       +1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

May 14 '24 14:05 codecov[bot]

I notice the ! in the PR title, but I cannot spot what this breaks. I'm missing something.

May 15 '24 23:05 krancour

I notice the ! in the PR title, but I cannot spot what this breaks. I'm missing something.

I had to think again about why I thought this was breaking, and initially thought it wasn't, but then recalled the tiny difference.

At present, include and exclude paths for Git subscriptions look at the full diff between $previousCommit and HEAD. If this contains changes to included paths, the current HEAD is greenlighted to create Freight for.

With this pull request, this behavior changes, as we look at every individual commit to see if it contains changes that make it eligible to be used as an artifact for Freight. As the current HEAD may not contain an included path, this means that the newest discovered artifact can change compared to the latest Freight we created (before upgrading), causing the Warehouse to create new Freight with the exact commit that contained these changes (instead of HEAD).

May 16 '24 12:05 hiddeco

causing the Warehouse to create new Freight with the exact commit that contained these changes

Ah. That makes sense. I doubt many people will notice this very slight behavior change and if anyone does, I think they're likely to find the new behavior make more intuitive sense.

Fine with me if you want to put the ! back in the title so we don't forget about this when reviewing commits to produce v0.7.0 release notes. It may prove to not be worth calling attention to, but at least we won't forget about it.

May 16 '24 13:05 krancour