An old build system started running again and downgraded the OS and remoting version on a number of images

[css-inverso]

Jenkins and plugins versions report

Environment

not applicable

What Operating System are you using (both controller, and any agents involved in the problem)?

We use the docker-inbound-agent as basis for some of our images.

Reproduction steps

• start 4.13.2-1 in docker and attach to it in a shell
• cat /etc/os-release

:~# docker run -it --user root --rm jenkins/inbound-agent:4.13.2-1-jdk11@sha256:620a26e4729b88076a3222a44501b96a
03dfbc08937585c7562d9e85bf3be6ce /bin/bash

Expected Results

:~# docker run -it --user root --rm jenkins/inbound-agent:4.13-2-jdk11@sha256:1cc07036180e1f2b001be7be1bdf57ba36bfbd0e1fb2103dfab8ee1464c5aa85 /bin/bash
Unable to find image 'jenkins/inbound-agent:4.13-2-jdk11@sha256:1cc07036180e1f2b001be7be1bdf57ba36bfbd0e1fb2103dfab8ee1464c5aa85' locally
docker.io/jenkins/inbound-agent@sha256:1cc07036180e1f2b001be7be1bdf57ba36bfbd0e1fb2103dfab8ee1464c5aa85: Pulling from jenkins/inbound-agent
dbba69284b27: Already exists
518e9c95b7a9: Pull complete
a95f8d94c795: Pull complete
5bdbc8307ec4: Pull complete
8d777ae9e3ef: Pull complete
89912d326a4e: Pull complete
4f4fb700ef54: Pull complete
05f16a9776d8: Pull complete
8b19489d2afd: Pull complete
Digest: sha256:1cc07036180e1f2b001be7be1bdf57ba36bfbd0e1fb2103dfab8ee1464c5aa85
Status: Downloaded newer image for jenkins/inbound-agent@sha256:1cc07036180e1f2b001be7be1bdf57ba36bfbd0e1fb2103dfab8ee1464c5aa85
root@57156ed09ab2:/home/jenkins# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Actual Results

Unable to find image 'jenkins/inbound-agent:4.13.2-1-jdk11@sha256:620a26e4729b88076a3222a44501b96a03dfbc08937585c7562d9e85bf3be6ce' locally
docker.io/jenkins/inbound-agent@sha256:620a26e4729b88076a3222a44501b96a03dfbc08937585c7562d9e85bf3be6ce: Pulling from jenkins/inbound-agent
0bc3020d05f1: Pull complete
2d902f15c74b: Pull complete
ec6bd7744224: Pull complete
ef9c22ebf313: Pull complete
52e8e308f6eb: Pull complete
6fc4099ca080: Pull complete
d2c51effb7dd: Pull complete
c938c96efd9d: Pull complete
e6321e72a6a5: Pull complete
b0b2a8516778: Pull complete
Digest: sha256:620a26e4729b88076a3222a44501b96a03dfbc08937585c7562d9e85bf3be6ce
Status: Downloaded newer image for jenkins/inbound-agent@sha256:620a26e4729b88076a3222a44501b96a03dfbc08937585c7562d9e85bf3be6ce
root@e22870e6cef8:/home/jenkins# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Anything else?

No response

[MarkEWaite]

Thanks for reporting the issue. I can duplicate the issue. Not yet sure of the root cause, but I am surprised at the output;.

[css-inverso]

From a casual glance I think that the docker-agent version in the Dockerfile and the image is actually build with the value stated in the Dockerfile (jenkins/agent:4.9-1-jdk11) is used. And that image is a buster one. As for why that would happen now and not in previous builds: no idea.

[dduportal]

Thanks for reporting! I can also replicate, we have to fix this.

It seems that there were "DockerHub Automatic Builds" enabled for this repository during the past few days (thanks @timja for catching this).

From a casual glance I think that the docker-agent version in the Dockerfile and the image is actually build with the value stated in the Dockerfile (jenkins/agent:4.9-1-jdk11) is used. And that image is a buster one. As for why that would happen now and not in previous builds: no idea.

That is true only if the build process does not provide a custom build-argument values: The official deployment process overrides with the latest detected version: https://github.com/jenkinsci/docker-inbound-agent/blob/master/Jenkinsfile#L95 , which overrides the default at https://github.com/jenkinsci/docker-inbound-agent/blob/master/docker-bake.hcl#L40-L42 (used for ci or local builds).

But the "DockerHub Automatic Build" does not, which seems to confirm the initial assumption.

I'm checking the deployment build to see if we can override these faulty images

[dduportal]

Rebuilt the tag and latest, it seems fixed on my local test:

$ docker pull jenkins/inbound-agent:4.13.2-1-jdk11 && docker run --rm --entrypoint=cat docker.io/jenkins/inbound-agent:4.13.2-1-jdk11 /etc/os-release
# ...
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

[dduportal]

@css-inverso can you confirm by closing or commenting the issue? Many thanks for reporting!

[conf]

Could you please also fix moving tags latest, jdk11, latest-jdk11 and others, they report older version of jenkins-agent (4.10) when the latest seems to be 4.13.2?

How to reproduce:

docker run -it --rm --pull=always jenkins/inbound-agent:latest -version
latest: Pulling from jenkins/inbound-agent
Digest: sha256:4eb482648dfc98ae59e9b62bdd5d4c1bef5a50a8fb6ecd1fb264ab03fe70beff
Status: Image is up to date for jenkins/inbound-agent:latest
4.10

[dduportal]

Closing as it was fixed: see https://github.com/jenkinsci/docker-inbound-agent/issues/279

[dduportal]

Reopening as old tags are still being pushed (as identified by @sbollap1 in Gitter) :

https://hub.docker.com/layers/jenkins/inbound-agent/4.10-3-jdk11/images/sha256-6d2c6b2fe3d7e1688f0e7edbf2200d3c67cf4cbea458fd832467d39a86ddb97f?context=explore

says "pushed 2 days ago" while the tag is 1+ year old

[dduportal]

Closing the issue as we were able to locate the "old system": it was the DockerCloud/DockerHub system: some remnants from the pasts. We deleted the SSH deploy keys and webhooks from jenkins/docker-agent, jenkins/docker-inbound-agent and jenkins/docker-ssh-agent to ensure that the "latest" tags are not overriden by unplanned pipelines.