Support Cloud Native Buildpack Artifacts

[joeybrown-sf]

Cloud Native Buildpack Artifact Types

Cloud Native Buildpacks (CNB) is a spec-driven project used for transforming source code to OCI Images. At the time of writing, CNB is a CNFC incubating project.

Developers and platform operators interested in using CNB should have a great discoverability story for finding and learning more about two artifact types:

• CNB Component Buildpacks
• CNB Builders

In service to that goal, this is a proposal for Artifact Hub to support these artifact types.

Artifact Types

Component Buildpacks

Component Buildpacks are often referred to as simply "buildpacks".

What is a buildpack?

Component buildpacks are distributed as OCI artifacts that contain metadata and executables used for the CNB build and detect processes.

Component buildpacks are typically used to build language-specific applications. There are many community buildpacks such as heroku buildpacks and paketo buildpacks. A reference to the CNB registry, which indexes community buildpacks, is provided later in this document.

Builders

What is a builder?

Builders are distributed as OCI Images that contain metadata, component buildpack layers, and a lifecycle executable (used to assemble the end-result application OCI Image).

Builders are typically used by platform operators that support a number of languages in which applications can be built. For example, if a platform supports python and node, a Builder will contain python and node buildpacks. A platform typically creates its own Builder(s) that may be of use to the community. For example, here are the Heroku Builders and the Paketo Builders.

Discoverability Today

CNB runs a bespoke, centralized registry for community buildpacks today, hosted at https://registry.buildpacks.io/. This registry provides basic discoverability. For example, see the search results for python buildpacks. This registry does not host the artifacts themselves. It serves as an index which refers to the registry that hosts a buildpack. This example demonstrates a buildpack where some versions are hosted on gcr and some versions hosted on docker.io.

This registry is utilized by the CNB cli tool, pack, as one method of exploring, discovering, and resolving buildpacks.

Community buildpacks are indexed via the registry-index project. Users may add community buildpacks by submitting an issue to the registry index project via the pack tool. Here is an example. The existing registry-index has an ownership model for component buildpacks, outlined at the bottom of this page and illustrated by this example.

The registry api (https://registry.buildpacks.io/) utilizes the registry-index as the source of truth for community component buildpacks. This API is used by pack.

This registry supports component buildpacks and not builders.

Artifact Hub Integration

It would improve the CNB artifact discoverability story to define and integrate Component Buildpacks and Builders. My understanding is that Artifact Hub exists to support this kind of mission.

As mentioned, CNB has an index for community Buildpacks, but not an index for Builders.

Questions

Does this seem like a good fit for Artifact Hub?

Can we continue using the existing CNB community buildpack registry? Perhaps we could create a specialized tracker source for component buildpacks? Does it make sense for CNB to create a registry/index for builders before integrating with Artifact Hub?

Does CNB need to host an instance of Artifact Hub from which ArtifactHub.io pulls?

Closing Thoughts

Before integrating with Artifact Hub, we would want sign-off on the CNB side. CNB is a spec-driven project and decisions like this are generally discussed and approved through a formal RFC process. Here is a CNB RFC for integrating with Artifact Hub. It is a concept of a plan. When we are in agreement about specifics of what must be done in order to support these new Artifact Hub artifact types (and I understand more about Artifact Hub), I can finish up that CNB RFC and work to get it through the approval process.

We are interested in integrating CNB artifacts with Artifact Hub, but we don't want ArtifactHub.io to be a hard dependency for the pack tool. Perhaps an "extension" of the CNB project would be how we would like to think about it.

I would love to get some feedback from yall on this request, and I'm happy to answer questions yall may have.

Thank you for your time and consideration!

[tegioz]

Hi @joeybrown-sf 👋

Thanks for this great introduction!

We'll go through the links you shared to get a better understanding of the CNB project, and we'll get back to you soon (probably later today or tomorrow, as soon as we catch up with a few things) 🙂

[tegioz]

Hi!

Both CNB buildpacks and builders would be a nice addition to Artifact Hub 🙂

Let me first explain how Artifact Hub indexes content.

How Artifact Hub works

Any organization or user can add repositories of any kind to Artifact Hub (AH). A repository in AH is like a metadata catalog that describes one or more packages of the same kind (Helm charts, or CNB buildpacks, for example). Repositories are polled periodically and metadata is collected from them as needed.

Note that once we've added support for a new artifact kind (i.e. buildpacks), anyone would be able to publish artifacts of that kind in AH. The CNB project could setup its own AH repository (or repositories) if they'd like. But the important point here is that there will probably be CNB artifacts published in AH that may not be in the current CNB registry. This process is fully automated and does not require any explicit approval or action on our side.

When using the generic tracker [1], AH repositories (metadata catalogs) are expected to be hosted in a git repository. We use a flexible directory structure that supports one or more packages per repository, including multiple versions per package if needed. Each package version must include a metadata file. Data unique to the artifact kind can be added in the form of custom annotations.

[1] Most of the artifact kinds supported, and all the recent additions, rely on this tracker. The main reason is to keep AH as simple as possible to maintain and extend, as well as to provide a unified user experience for users to publish content of any kind on AH.

Current registry

One of the first decisions the CNB project will need to make is if they'd like to keep the current registry or not.

Some projects, like Helm, shut down their own hub and moved to Artifact Hub. They used to maintain a repository with lots of charts, similar to your current registry repo. Maintaining it required a lot of work, and they switched to a more decentralized approach where each publisher was in charge of publishing their own repositories in AH.

If you decide to keep it, you'll need to consider if you'd like to keep them in sync, and how. You could try to automatically generate an AH repository from the content of your registry (by generating the required directory structure and metadata files). Then, you could publish that repository, and everything available in your registry would be listed in AH. But going this way would have some limitations [2] and require some extra work on your side to build and maintain that generation tool. Syncing the other way around (remember that anyone could add more CNB artifacts to AH that wouldn't be in your registry) would require more work on your side: pull from the AH API, add to your registry, handle potential conflicts/duplicates, etc.

Depending on which decision you make regarding the current registry, we may be able to help with the integration with the pack tool as well, so that it can search for CNB artifacts on AH. The Helm CLI tool relies on the AH API for searches too, for example. We'd be happy to provide a dedicated API endpoint if needed. But you mentioned that you didn't want AH to be a hard dependency, so not sure if this is something you'd be interested in.

[2] Publishers not being able to make the most of all the AH features due to the lack of direct access to the metadata file, the CNB project receiving notifications for errors from AH in any of the packages in the repo -instead of the original publisher-, etc.

Questions

Does this seem like a good fit for Artifact Hub?

Of course!

Can we continue using the existing CNB community buildpack registry?

You could, but there are some points to keep in mind as I mentioned in the current registry section.

Perhaps we could create a specialized tracker source for component buildpacks?

Unless strictly necessary, we are trying to rely on the generic tracker source for new artifacts supported. The main reason is to keep AH as easy to maintain as possible (we support 25+ artifact kinds!). We also think it's important to have a common user experience when publishing content on AH. Using the same format makes everything simpler for users as well, as it's the same for them to setup a repository with Kubewarden policies than one with CNB buildpacks. They can also reuse the same tooling or automation processes they may have.

Does it make sense for CNB to create a registry/index for builders before integrating with Artifact Hub?

I don't think it would help us with the integration, so probably only if you decide to keep your registry.

Does CNB need to host an instance of Artifact Hub from which ArtifactHub.io pulls?

No, don't worry, you'd be using the instance at https://artifacthub.io.

Hope this helps 🙂

[joeybrown-sf]

Thank you so much for this thorough response @tegioz! I will put my thoughts together and respond soon. This is very helpful!

[joeybrown-sf]

I am working on the CNB RFC and I have questions about repositories and ownership claims. For context to my questions, I am exploring if we could completely replace the CNB registry in favor of ArtifactHub.io, which is contrary to a point I previously made ("we don't want ArtifactHub.io to be a hard dependency").

I am thinking that CNB registry operators may be able to facilitate a migration to ArtifactHub.io by registering buildpacks in ArtifactHub.io that are already indexed in the buildpacks registry.

Here is an excerpt from the CNB RFC that I have not published yet. I wanted to discuss and make sure my understanding is correct. This excerpt is part of a potential migration story. My questions are after this excerpt.

Establish a CNB organization in ArtifactHub.io. This organization will be the ArtifactHub.io package owner for these packages until publishers claim ownership themselves via the ArtifactHub.io control panel.

In order to mirror the buildpack artifacts to ArtifactHub, the CNB registry index repository will set up a directory structure that contains a metadata file for each package version. For reference, please see the Inspektor Gadget repository structure.

In ArtifactHub.io, anyone is able to publish any packages. The CNB organization will create package metadata for use by Artifact Hub. These packages will be created for existing packages and any new buildpacks published to the buildpack registry.

The file structure in the buildpack registry index will look something like this:

path/to/packages
├── heroku
│   ├── dotnet
│   │   ├── 0.1.0
│   │   │   └── artifacthub-pkg.yml
│   │   ├── 0.1.1
│   │   │   └── artifacthub-pkg.yml
│   │   ├── 0.1.2
│   │   │    └── artifacthub-pkg.yml
│   │   └── ...
│   └── java
│       ├── 0.1.1
│       │   └── artifacthub-pkg.yml
│       └── ...
├── initializ-buildpacks
│   ├── go
│   │   ├── 1.0.0
│   │   │   └── artifacthub-pkg.yml
│   │   ├── 1.0.1
│   │   │   └── artifacthub-pkg.yml
│   │   ├── 1.0.2
│   │   │    └── artifacthub-pkg.yml
│   │   └── ...
│   ├── php
│   │   ├── 1.0.0
│   │   │   └── artifacthub-pkg.yml
│   │   └── ...
│   └── ...
├── paketo
│   ├── python
│   │   ├── 0.0.1
│   │   │   └── artifacthub-pkg.yml
│   │   ├── 0.1.0
│   │   │   └── artifacthub-pkg.yml
│   │   ├── 0.2.0
│   │   │    └── artifacthub-pkg.yml
│   │   └── ...
│   └── ruby
│       ├── 0.4.0
│       │   └── artifacthub-pkg.yml
│       └── ...
└── ...

Here is my question. When using the Artifact Hub control panel to add a repository, I see a URL section. Can this URL include relative paths? In other words, when adding a repository, could I add the following repository URLs as separate Artifact Hub registries?

• https://github.com/buildpacks/registry-index/path/to/packages/heroku/dotnet
• https://github.com/buildpacks/registry-index/path/to/packages/heroku/java
• https://github.com/buildpacks/registry-index/path/to/packages/initializ-buildpacks/go
• https://github.com/buildpacks/registry-index/path/to/packages/paketo/python
• https://github.com/buildpacks/registry-index/path/to/packages/paketo/ruby

I think if we can do this, then publishers will be able to claim ownership of the different repositories. My main question is this: Is there a 1:1 relationship between Artifact Hub repositories and git repositories, or is it possible to have multiple Artifact Hub repositories within a single git repository?

Forgive me if I am overlooking something obvious.

[tegioz]

When using the Artifact Hub control panel to add a repository, I see a URL section. Can this URL include relative paths?

Yes!

In other words, when adding a repository, could I add the following repository URLs as separate Artifact Hub registries?

Yes, that sounds great! (without the ˋpath/to/packagesˋ please, that is like a placeholder for ˋheroku/dotnetˋ).

I think if we can do this, then publishers will be able to claim ownership of the different repositories.

Great approach, love it! 😍

When they claim the ownership of the AH repo, they can always replace the url with another location they've full control of. And you'd be able to remove that part of the tree from your git repo.

We'd really appreciate if you could add a readme file to packages, even if it's a small one. It can be of great help for users (specially newcomers to the artifact kind) and it makes packages look much nicer 😇

You can play a bit with https://staging.artifacthub.io if you need to run some tests.

[joeybrown-sf]

Perfect. Thank you!

[joeybrown-sf]

Thank you for your feedback on this, @tegioz. I've updated the CNB RFC with a concrete proposal. I'd love for you to take a look when you get a chance and I welcome your input and feedback.

There is a CNB community meeting on Thursday at 12:00PM EST. I've put ArtifactHub.io integration as a discussion topic on the agenda. I don't know how long we will focus on this and I cannot guarantee a large turnout at this meeting. That being said, your presence is not necessary but it is welcome! If you cannot make this one, we'd love to have you any time!

Once we reach consensus on the CNB proposal, I will leave a comment on this Issue letting you know whether this integration is something the CNB team would like to pursue. If so, you can make your decision on whether or not to accept this proposal for CNB artifact types.

Thank you sincerely for your time and effort on this proposal feedback and thank you for your work on Artifact Hub!

[tegioz]

No worries! I'll try to take a look at it later today.

We'd be happy to add support for CNB artifacts once you've signed-off on your side 🙂

Thank you!

[tegioz]

⚠ off-topic ⚠ 😅

Not sure how you handle voting in the CNB rfcs repo @joeybrown-sf, but GitVote could be a nice fit. Just in case it helps 🙂

[tegioz]

The CNB RFC looks great @joeybrown-sf 👍

For instance, the pack buildpack register command will call the ArtifactHub.io API

I may have misunderstood this, but the comment below only applies if you were thinking about allowing users to publish content on AH using the pack tool

Some thoughts about this.

In order to publish a repository on AH, users need to sign up and log in first. They could create an AH API key from the control panel and expose it to pack via environment variables. Note that we don't host the artifacts themselves, so they'd still need to be published to a registry.

In addition to this, it looks like pack buildpack register works at the buildpack level. Even if you were automating the publication of artifacts on AH using the API, you'd only need to setup the AH repository once, not per buildpack. To add a buildpack to an existing repository, just the package metadata would need to be added to the git repository, which wouldn't require talking to the AH API. Not sure if adding that metadata to a git repository is something you're planning on doing from pack too. IMHO it'd be just easier if the task of publishing content on AH was handled separately, not from pack. It's usually a very straightforward process and we have some documentation for each artifact kind supported.

Ownership claim

Something I forgot to mention about ownership claim. The process relies on the artifacthub-repo.yml metadata file. We check that the user claiming the ownership is listed in the owners section. When the user has control over the git repository, they just need to add the metadata file listing themselves as owners. We consider that a proof of ownership and we proceed with the transfer (it's all automated). In your case, it'll be a bit different. To make the process work, a repository metadata file will need to be added to the corresponding directory in your git repo (i.e. one for Heroku). It'll need to include the emails of the users you authorize to claim the repository's ownership (the same email they use on AH). This could be done in advance or when they are ready to claim the ownership of a repo (by summiting themselves a PR with the repo metadata file).

Hope this helps. Will let you know if something else comes to mind 🙂

[joeybrown-sf]

Hey friends, I just wanted to let you know this has not stalled out and is under active discussion. I'm putting together some diagrams in the Buildpacks RFC to help illustrate and compare current workflows with the future workflows for buildpack publishers. Once I've got that pushed up, we should be quite close to acceptance.

Thanks again for all your help!