private docker image registry for staging core security releases

[jtnord]

Service(s)

Other

Summary

during security upgrades ci.jenkins.io is taken offline at the start of the process and then brough online at the end.

The server could be made available at the start if there was a private container repository that could be used for staging images (it would also mean the images can be built in advance).

At the time of writing this ci.jenkins.io has been down for about 3 hours and I am unable to get to test results etc to investigate failures on plugins and the like.

This was apparantly requested by @daniel-beck ~2018 but I could not find any ticket for this.

Reproduction steps

1 start a jenkins security release 2. go to ci.jenkins.io

expect that in 2 the server is down only for as long as it takes to reconfigure an image and start. actual Jenkins is down for multiple hours.

[timja]

see also: https://github.com/jenkinsci/docker/issues/1228

[daniel-beck]

This was apparantly requested by @daniel-beck ~2018 but I could not find any ticket for this.

IIRC (and it's been a while, so I may well misremember!), I requested this from Olivier when he presented his design for the automated core CI in the CloudBees San Jose office.

(FTR today's advisory isn't core, and the delay has other reasons. I accidentally misled @jtnord when I answered a question about today's downtime, sorry about that.)

[dduportal]

It would, indeed, be a really good improvement! 👍

A few thoughts I collected since 2021 and the initial request:

• I believe we have to separate the "Staging jenkins/* Docker images" which are consumed by Jenkins user, from the "Staging the infrastructure image" we use for running ci.jenkins.io.

• A production service such as ci.jenkins.io is managed by infrastructure as code. It means we have to change its image from "jenkins/jenkins:lts-jdk21" today to fixed versions (using digest and/or tags. I prefer having both ideally).

  • Note that the current "latest" pattern lts-jdk21 is still there because we (jenkins infra team) never had/took the time to change it at the risk of disrupting security advisories process. But we should (and be pro-actives to propose JenSec an update on their process).
  • The "staging image for ci.jenkins.io" could also have staged plugins in it. It would cover cases such as today (e.g. no Core, but Plugins).
  • The only thing still unclear to me (like it was for Olivier in the past) is how do we manage the image update for JenSec team in autonomy, since our code is public. We can eventually encrypt the tag value/docker registry or image name and ask them for a PR when they have published (eventually an automated PR with a human, or automatic commit)..

• For the jenkins/* images, it's a really easy one technically:

  • Infra team sets up a private container registry close to trusted.ci (where publication happens) => we already have an Azure Container Registry used for caching that can be reused for private namespacing.
  • Security may have a "build&publish" security pipeline, with same code as the current "build& publish" but with the private credential registry + an env var specifying the container registry, thanks to Docker Bake Setups.
  • Security also need a "promote staging to production" pipeline which would use container tool to copy layers from private registry to DockerHub when they run the publication. skopeo CLI is clearly the best candidate here (efficient, no need for container engine, etc.).
  • I believe the whole "build, publish, promote" pattern should be the default (so we reuse almost all the code to avoid drift when running security advisory)

[daniel-beck]

Could we not publish Docker images to repo.jenkins-ci.org (private repo), and then promote them to Dockerhub on release day? That would allow Jenkins infra to consume private staged artifacts in advance of publication. The only complication would be to implement the startup/pull script so that it obtains the newest tagged image from either repo, unless there's some support for this in Docker directly. (I'm not against specifying hashes or non-mutable versions either -- it'll just be another PR similar to what we already do in jenkins-infra/release. These just seem mostly independent problems.)

For Maven artifacts, we do this similarly, and it's not a big deal that people know that repo.jenkins-ci.org/pitty exists, they cannot access it.

[dduportal]

Could we not publish Docker images to repo.jenkins-ci.org (private repo), and then promote them to Dockerhub on release day? That would allow Jenkins infra to consume private staged artifacts in advance of publication. The only complication would be to implement the startup/pull script so that it obtains the newest tagged image from either repo, unless there's some support for this in Docker directly. (I'm not against specifying hashes or non-mutable versions either -- it'll just be another PR similar to what we already do in jenkins-infra/release. These just seem mostly independent problems.)

For Maven artifacts, we do this similarly, and it's not a big deal that people know that repo.jenkins-ci.org/pitty exists, they cannot access it.

I mentioned private registries out of caution as they are only reachable through private networks (for both trusted and ci.jio).

But yes, that would be totally possible to use Artifactory for pulling images:

• A service account credential will be needed and passed to Docker engine on the ci.jenkins.io VM (through our infra. as code system) so that Docker can pull the image
• The REGISTRY build variable for publication would be set to something like repo.jenkins-ci.org/xxxx/jenkins/jenkins:yyy.

For the updates, provided a read-only Artifactory token, our tool updatecli is expected to be able to get the latest versions.

[timja]

Keep this in mind: https://github.com/jenkinsci/docker/issues/1228#issuecomment-961783533

you would need to get a SAN on the certificate on docker hub and then continue to provide that whenever the certificate is updated.

[dduportal]

Keep this in mind: jenkinsci/docker#1228 (comment)

you would need to get a SAN on the certificate on docker hub and then continue to provide that whenever the certificate is updated.

Yup, thanks for the reminder. It's an easy thing nowadays: since a few month we have access to the JFrog portal which allows us to set up certificates. So adding a SAN looks doable.

[timja]

Ah ok, that's much easier than before.