feat(warehouse)!: artifact discovery and manual Freight creation

[hiddeco]

TL;DR

This pull request addresses the majority of the backend part of #761, by reflecting the last 20 discovered artifacts for each subscription of a Warehouse in its Status, allowing the UI to present the user with options to pick from to "mix-and-match" their selection of artifacts to create Freight for.

Combined with these changes, a "manual" and "automatic" Freight creation mode has been introduced to the Warehouse, in addition to changes to the reconciliation model, and further optimizations to the logic that gathers information about upstream artifacts.

API changes

.status.discoveredArtifacts

For each subscription of a Warehouse, Kargo will discover the 20 latest artifacts based on the selection strategy and reflect them in .status.discoveredArtifacts field of the object. The UI can use this information to provide the user with the option to "mix and match" artifacts for which to create Freight.

An example:

discoveredArtifacts:
  charts:
    - repoURL: oci://ghcr.io/akuity/kargo-charts/kargo
      versions:
        - 0.6.0
        - 0.6.0-rc.2
        - 0.6.0-rc.1
        # ...omitted for brevity
  commits:
    - repoURL: https://github.com/akuity/kargo.git
      commits:
        - author: Remington Breeze <[email protected]>
          branch: main
          createdAt: "2024-05-15T18:58:00Z"
          id: c5c74d42069b41461763bc3c21de02b579ada1dc
          subject: 'fix(ui): Promotion confirmation message overflowed freight box (#2015)'
        - author: Remington Breeze <[email protected]>
          branch: main
          createdAt: "2024-05-15T18:55:00Z"
          id: f65a27fcb5bc48861d10768313bb5e4201a8cb86
          subject: 'feat(ui): Ability to abort verification from UI (#2013)'
        # ...omitted for brevity
  images:
    - repoURL: ghcr.io/akuity/kargo
      images:
        - digest: sha256:eae672841a1756683a967b7c25eab31d694ac66ecd09b35191664ec166530f1a
          tag: v0.6.0
        - digest: sha256:79103eb0596ac4c6e12f2eedd8e9066d8557ff86ba58b9a4037d5bdbd74c1ac0
          tag: v0.6.0-rc.2
        - digest: sha256:0ee72b9e8955a7bf4f4b78667ebe9612983e3d0d184eab14f75b1b87c3de31d9
          tag: v0.6.0-rc.1
        # ...omitted for brevity

Note: Not all fields for images and commits are shown above, as the availability of data (e.g. a Git tag, or the creation timestamp of a container image) depends on the chosen selection strategy. Please refer to the in-code changes for further details on all fields.

.spec.freightCreation

This newly introduced field configures if Freight is automatically created for newly discovered artifacts. It can be set to Automatic (the default) to discover artifacts and automatically create Freight for the latest discovered version(s), or to Manual to discover artifacts without creating Freight.

Behavioral changes

• Warehouse reconciler will first discover the latest artifacts for each subscription, and then (optionally) automatically create Freight for it.
• The discovery of Git artifacts required having access to tree or repository history in many cases. Because of this, and to continue to keep things performant, the cloning process has changed from using a "shallow clone" (--depth=1) to the newer "partial clone" (--filter) option. To read more about this, please refer to this excellent article.
• Git metadata of tags and commits is now gathered using git for-each-ref and git log with specifically designed custom formatters. Instead of performing FS-heavy git checkout, etc. operations.
• Instead of looking at the diff for all commits between the current Freight and HEAD to determine if it contains changes to included and/or excluded paths. Every individual commit is inspected until 20 matching commits are found (or no commits remain).

Other notable changes

• For commits, we do now take note of the author.