Publishing Kubernetes packages on community infrastructure

[justaugustus]

Enhancement Description

• One-line enhancement description (can be used as a release note): Publishing Kubernetes packages on community infrastructure
• Kubernetes Enhancement Proposal: keps/sig-release/1731-publishing-packages/README.md
• Discussion Link: https://docs.google.com/spreadsheets/d/1Vgpb7HPg8WgoFtDuddaRD6j7jQLoqNK6vUu6Lb0ea28
• Primary contact (assignee): @justaugustus @saschagrunert
• Responsible SIGs: SIG Release
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): v1.28
  • Beta release target (x.y):
  • Stable release target (x.y):
• [ ] Alpha
  • [x] KEP (k/enhancements) update PR(s): https://github.com/kubernetes/enhancements/pull/843 https://github.com/kubernetes/enhancements/pull/3434 https://github.com/kubernetes/enhancements/pull/3750
  • [ ] Code (k/k) update PR(s):
  • [ ] Docs (k/website) update PR(s):

---

Milestones and Tasks Checklist

Milestone 1.0—Code Deliverable

• [ ] Success looks like: debs and rpms are in multiple locations (Google infra and third-party hosting service(s)) @detiber
  • [ ] Define User Scenarios and Requirements (Configuration, management, monitoring, etc.)
  • [ ] Describe proposed Flow (what are key actions that the user will take? How will they take them?)
  • [ ] Choose the third-party hosting service/Set up a package host of some sort @detiber, @leonardpahlke helping
    • [x] Identified some options to explore
    • [ ] Outreach to reps of those options to see if they'll work with us on this WIP
  • [ ] Do a spike test to see if the third-party hosting service meets our needs @detiber, @leonardpahlke helping
  • [ ] Publish the debs/rpms to the third-party hosting service @detiber

Milestone 1.0—Documentation Deliverable

• [ ] Success looks like: Entire deb/rpms move to community infra plan is outlined in a KEP
  • [x] Update this KEP: @RobertKielty (made a PR July 15, 2022)
  • [x] Review the KEP @saschagrunert, @ameukam, @detiber
  • [x] Approve the KEP
  • [x] Get an understanding of the top-level script: what type of infra we're going to run, who pays for this, who can access to this, what type of account we have, tools available @RobertKielty - Script reviewed by Robert Kielty and discussed on SIG Release channel with Ben Elder. This KEP will drive outstanding infra/billing questions for the new implementation.
  • [x] Clarify what "top-level script" means (linked in cell above) @RobertKielty
  • [ ] Finalize defining the user personas for this work, with group review and sign-off @detiber, @LappleApple

Milestone 1.0—Risk Mitigation

• [ ] Success looks like: All risk matters for this milestone are accounted for, plans in place to mitigate
  • [ ] Risk: A new GPG key will be issued incompatible with the existing GPG key managed by Google. This is a breaking change for the existing infrastructures using DEB/RPM packages as installation artifacts

Milestone 1.0—Questions resolved

• [ ] Resolve if this is needed: Give access to the sandbox env to release eng to work on it @ameukam, @onlydole helping
• [ ] Develop sandbox env (GCP project) and give access to Rel Eng to work on it @ameukam, @onlydole helping
• [x] Needed more discussion: Meeting to specifically discuss if we are doing GPG or if we are owning the packages; then who owns new GPG keys and how and when we store them. @ameukam, @onlydole helping
• [ ] Generate new keys : https://github.com/kubernetes/release/issues/2627
• [ ] Decide on plan: Be able to issue new gcp trusted keys for entire ecosystem/whoever is relying on those packages to deploy/install/upgrade K8s project
• [ ] Find/line up people who can build packages for debian, CentOS to work on build tooling (@upodroid to help build Debian packages)

Milestone 2.0—Code deliverable

• [ ] Success looks like: implementation (source code) in our repositories, such that we can publish releases by ourselves without having to rely on Google people at all
  • [ ] Define User Scenarios and Requirements (Configuration, management, monitoring, etc.
  • [ ] Proposed Flow (what are key actions that the user will take? How will they take them?)
  • [ ] Refine the current debs/rpms built by kubepkg
  • [ ] Resume discussing how we rethink / rewrite the build tools
  • [ ] Rebase the existing kubepkg work into something krel can use directly
  • [ ] Krel, the kubernetes release tool, should start including a package build step running the build scripts
  • [ ] Phase out publishing to the google controlled host, for having krel publish to our new host for 1-2 releases
  • [ ] Users migrate from the Google registry to the community registry, adding the signing key

Milestone 2.0—Documentation Deliverable

• [ ] Success looks like: we've published docs on how process works and when it's not working as expected, what to do (for release managers)
  • [ ] Expand documentation on the top-level script to drive knowledge transfer
  • [ ] Expand documentation on the package build scripts for Debian and RPM to drive knowledge transfer

Milestone 2.0—Questions resolved

• [ ] Success looks like: @saschagrunert and @justaugustus have provided context and knowledge about the package build scripts for Debian and rpms
  • [ ] Answer: what type of infra we're going to run
  • [ ] Answer: who pays for it
  • [ ] Answer: what type of account do we have
  • [ ] Answer: what tools do/will we have available

Milestone 3.0—Documentation deliverable

• [ ] Success looks like: We've updated the google build+publish+release script to download packages uploaded by the upstream release process

Milestone 4.0— Documentation Deliverable

• [ ] Success looks like: We've published docs for end users about old infra and new/migrated (more static)
  • [ ] Write email to community regarding the required migration to the community repository
  • [ ] Send email to community regarding the required migration to the community repository
  • [ ] SIG Release informs the community so we don't break people over just one milestone: "these are the keys we'll use to release 1.2X..."

---

Why is this needed?

• It's part of the effort for the community to fully own the infrastructure related to the Kubernetes project.
• Ensure all aspects of project releases can be staffed across companies
• We don’t want to be dependent on a Build Admin any longer (which tends to slow releases)
• We seek more control to eventually choose to build packages for prereleases, extend what packages are shipped, etc.

Why is it needed now?

• The existing infrastructure of this relies on a workstation inside Google offices—essentially one person. It’s not reliable or sustainable.

Who needs it? (User Personas WIP)

• Google Build Team as the existing team building the system packages.
• What does “done” look like/acceptance criteria?
• Automated builds of deb and rpm Kubernetes packages within community infrastructure

Related open issues

• #913

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

[k8s-triage-robot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[saschagrunert]

@RobertKielty @LappleApple feel free to propose an update to the existing KEP

[xmudrii]

Summary

We started evaluating OpenBuildService (OBS) hosted by openSUSE (https://build.opensuse.org/) as a solution for building, signing, publishing, and hosting our packages. OBS handles Debian (deb) and RPM packages for many of the most popular distros (Ubuntu, Debian, CentOS, RHEL...).

The initial proof-of-concept phase went very well:

• packages are being built successfully using the same spec files with no or very minor changes
• it's possible to install those packages and successfully provision a cluster
• it's possible to migrate to the new OBS repos and then upgrade a cluster

The initial proof-of-concept covered both debs and rpms on amd64.

Open questions

Several open questions are still to be investigated/tested or decided upon.

• How do we want to handle multi-arch builds?
  • Option 1: try to parametrize spec files to choose the correct binary from a tarball with all binaries for all supported architectures
    • Potential issue is that such a tarball might be big, but that might be mitigated by using a higher compression rate
  • Option 2: push sources to OBS and build binaries for the target architecture in OBS
    • There are three problems with this option:
      • Kubernetes sources are also big, ranging from 1 GB to 2 GB depending on if .git is included
      • Unless we get reproducible builds working in OBS, the resulting binary will be different than the binary we distribute via release buckets
      • The binary itself will not be signed (e.g. with cosign). This might not be a blocking issue because we sign the APT repository or the RPM package
• How do we want to organize our spec files?
  • OBS works in a way that you have a Project that holds packages. One package is published to multiple repositories and there's a repository for each supported/chosen operating system
  • For Debian packages, a package on OBS is needed for each Debian package -- you can't define multiple packages in one spec file
  • For RPM packages, one package on OBS can be used for building/hosting multiple RPM packages -- you can define multiple packages in the same spec file (that's what we do right now)
  • We have multiple options:
    • Option 1: continue what we're doing right now, however, having one single RPM spec is harder to maintain and harder for users to consume. We are also not consistent with debs
    • Option 2: break the current RPM spec into multiple specs (already done by kubepkg) -- this is currently the most favorable option
    • Option 3: use debbuild to build deb packages from RPM specs. The disadvantage is that deb packages are not "natively" defined in this case -- someone building deb packages might not have an idea bout how RPM works. This also introduces a new dependency
  • Note: we have the option to change our spec files, but we should try to keep the same results if that's possible -- so that users can migrate to the new packages without too much hassle

Community infrastructure concerns

The solution we're evaluating right now is hosted by openSUSE. openSUSE is willing to sponsor us for all our needs, so we can use their infrastructure. This is not "truly" community infrastructure, but this helps us a lot as we don't have to worry about building and publishing/hosting packages ourselves.

Additionally, OpenBuildService is an open source platform, so we can always decide to host it ourselves if we don't want to use openSUSE's infra for any reason. To make this possible, we'll look into building proxy/redirects from apt.kubernetes.io/yum.kubernetes.io to the openSUSE OBS infra.

Package/repository signing concerns

One of the biggest concerns we had is how are going to manage GPG keys for signing packages (in the case of rpm) and repositories (in the case of APT). The OBS platform fully manages the GPG keys -- the private key is not exposed to anyone using the platform. In those terms, we can look at OBS as an API that we can give access to without worrying that someone might get access to the key.

Moving building packages from Google to the community

Another concern is that we want to move building packages from Google to the community. We are planning to accomplish this by building packages on OBS and letting Google Build Admins download packages from there to be published to the Google (current) infra.

Next steps and timeline

The proposed timeline would be something along the following lines:

Pre-alpha

• Get answers to the open questions
• Extend the current proof-of-concept with multi-arch builds
• Final decision on do we want to proceed with OBS

Alpha

• Integrate OBS with the current release pipeline
  • Start building and publishing packages from OBS in parallel to doing that on the Google infra
• Work on tests to ensure the quality of created packages
• Communicate the progress, timelines, and upcoming changes to the community

Beta

• Publish packages built by OBS to the Google infra
• Another round of comms to the community

Stable

• Gradually phase out Google infra in favor of OBS

Alternatives considered

We considered building packages manually in our pipeline and publishing and hosting packages independently instead of using OBS. That however has much increased complexity, as we have to maintain the whole infra, and we also have to build a management system for GPG keys, which is very complex.

[saschagrunert]

• Get answers to the open questions

What do you think would be the best way to achieve this?

[saschagrunert]

We had a community meeting a few days ago to clarify the open questions. Updating the KEP will be one of the next steps here.

[xmudrii]

I'll take care of updating the KEP. /assign

[saschagrunert]

So I assume we will graduate this enhancement in v1.27.

[jeremyrickard]

/milestone v1.27

[jeremyrickard]

This is out of tree and isn't an end-user feature, so it doesn't need to have a PRR.

[shatoboar]

Hello @justaugustus 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for v1.27 (correct me, if otherwise)

Here’s where this enhancement currently stands:

• [X] KEP readme using the latest template has been merged into the k/enhancements repo.
• [X] KEP readme has a updated detailed test plan section filled out
• [X] KEP readme has up to date graduation criteria.

With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[shatoboar]

Hi @justaugustus, @xmudrii and @saschagrunert 👋 ,

Checking in as we approach 1.27 code freeze at 17:00 PDT on Tuesday 14th March 2023.

Please ensure the following items are completed:

• [ ] All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• [ ] All PRs are fully merged by the code freeze deadline.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze:

• https://github.com/kubernetes/release/pull/2946

Please let me know if there are any other PRs in k/k I should be tracking for this KEP. As always, we are here to help should questions come up. Thanks!

[xmudrii]

Hi @shatoboar,

This is a KEP for an out-of-tree change. As such, we currently don't plan on having PRs in k/k, only in k/release and other SIG Release repos. This also means that code freeze shouldn't affect us for this KEP.

https://github.com/kubernetes/release/pull/2946 is one of PRs that we need to merge, but we'll also have several other PRs in that repo.

[jeremyrickard]

/milestone v1.28

[ruheenaansari34]

Hello @justaugustus 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 1:00 UTC on Friday 16th June 2023.

This enhancement is targeting for stage alpha for 1.28 (correct me, if otherwise)

Here’s where this enhancement currently stands:

• [X] KEP readme using the latest template has been merged into the k/enhancements repo.
• [X] KEP status is marked as implementable for latest-milestone: 1.28
• [X] KEP readme has a updated detailed test plan section filled out
• [X] KEP readme has up to date graduation criteria.
• [X] KEP has a production readiness review that has been completed and merged into k/enhancements.

With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[Rishit-dagli]

Hello @justaugustus :wave:, 1.28 Docs Lead here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[xmudrii]

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

Yes. I'll follow up on this with folks and I'll make sure that we have everything in place by the deadline.

[ramrodo]

Hello @xmudrii, 1.28 Comms here.

Is this enhancement planned to have an opt-in process for Feature Blog delivery?

The deadline for opt-in is on July 19th, 2023, so please consider submitting a place-holder PR at kubernetes/website for this to be considered.

Thank you!

[xmudrii]

@ramrodo Is just creating a PR enough to opt-in or is there anything else that we should do?

[ramrodo]

@ramrodo Is just creating a PR enough to opt-in or is there anything else that we should do?

Yes. Creating the placeholder PR is enough to opt-in for now.

[Rishit-dagli]

@xmudrii

Yes. I'll follow up on this with folks and I'll make sure that we have everything in place by the deadline.

A reminder on this since there is 1 week to the deadline, this can even be a draft PR right now.

[xmudrii]

@Rishit-dagli Thank you, I'll make sure to create placeholders by the end of the week.

[xmudrii]

@Rishit-dagli @ramrodo I created placeholder PRs for both docs and feature blog: https://github.com/kubernetes/website/pull/42022 and https://github.com/kubernetes/website/pull/42023