infrastructure
infrastructure copied to clipboard
adoptium
Add Centos 6 docker build to centos7_docker_image_updater jenkins job
Currently our centos6 docker image is built and pushed from a github workflow, https://github.com/adoptium/infrastructure/blob/76d51f107ec24321ad498001d5b88474514518fe/.github/workflows/build.yml#L20
It would make sense to add this to our centos7_docker_image_updater jenkins job as part of the docker images it builds
NOTE: Any updates to this should be covered in a corresponding doc update to https://github.com/adoptium/infrastructure/blob/master/FAQ.md#what-about-the-builds-that-use-the-dockerbuild-tag
@Haroon-Khel @steelhead31 it looks like we have consistent failures on the job after the related PR was merged on the CentOS6 part. Can one of you take a look at this please as we need to be sure that when PRs are merged we verify that they don't cause failures.
Error log on latest build https://ci.adoptopenjdk.net/view/Failing%20Temurin%20jobs/job/centos7_docker_image_updater/331/execution/node/212/log/
+ docker push index.docker.io/adoptopenjdk/centos6_build_image:linux-amd64
The push refers to repository [docker.io/adoptopenjdk/centos6_build_image]
bc6a0138fc79: Preparing
513c80085c29: Preparing
54d345198942: Preparing
10506ac33692: Preparing
42efb8220fde: Preparing
14e739122a1b: Preparing
d4f51880a155: Preparing
aaa5621d7c01: Preparing
d4f51880a155: Waiting
aaa5621d7c01: Waiting
14e739122a1b: Waiting
denied: requested access to the resource is denied
Got a replica jenkins job which is just running the centos6 build at the moment https://ci.adoptopenjdk.net/job/centos7_image_update_haroon_test/8/console
Might be related to the fact that the first build and push requires the image to be tagged https://stackoverflow.com/questions/41984399/denied-requested-access-to-the-resource-is-denied-docker
Rerunning my replica jenkins job with a tag stage https://ci.adoptopenjdk.net/job/centos7_image_update_haroon_test/9/console
Still fails https://ci.adoptopenjdk.net/job/centos7_image_update_haroon_test/10/console
I'll try to replicate the build in an ssh session on the machine itself
While I am in the middle of finding the dockerhub credentials to be able to recreate the pushing of the image to dockerhub via the command line, I notice that without the credentials, I get the same error as above
jenkins@dockerhost-equinix-ubuntu2204-x64-1:~$ docker push index.docker.io/adoptopenjdk/centos6_build_image:linux-amd64
The push refers to repository [docker.io/adoptopenjdk/centos6_build_image]
a1c6beca1891: Preparing
c70d0fcd04cc: Preparing
ebb00c9d1a8e: Preparing
75f452af8a19: Preparing
bc0bb3a89452: Preparing
83c9603dc659: Waiting
d6d9e9e952b8: Waiting
aaa5621d7c01: Waiting
denied: requested access to the resource is denied
Even after logging into dockerhub I still get the same error
jenkins@dockerhost-equinix-ubuntu2204-x64-1:~$ dockerpush index.docker.io/adoptopenjdk/centos6_build_image:linux-amd64
The push refers to repository [docker.io/adoptopenjdk/centos6_build_image]
6d62c608a741: Preparing
b1280007a593: Preparing
a743d9d92074: Preparing
dbd761fa66a3: Preparing
aa6a55730597: Waiting
bec408233208: Waiting
e3d1bdc2e37f: Waiting
aaa5621d7c01: Waiting
denied: requested access to the resource is denied
Retagging the image does not solve the error, neither does building and pushing the image from the other x64 dockerhost machine dockerhost-equinix-ubuntu2004-x64-1
@Haroon-Khel , as mentioned on Friday :) , for both this one and the Risc-V push, I've ready a few articles on this issue, and it appears that this process sequence may work ...
- Build Image
- Docker Login
- Docker Push ( Expected to fail as above )
- Docker Logout
- Docker Login
- Docker Tag
- Docker Push
See this: https://stackoverflow.com/questions/41984399/denied-requested-access-to-the-resource-is-denied-docker
Update:
Scott's solution unfortunately did not work. Same denied: requested access to the resource is denied.
I suspect the error is to do with how the adoptopenjdk/centos6_build_image repository is setup. I have no problems pushing to the adoptopenjdk/centos7_build_image repo
Noting that while centos7_build_image was pushed by the adoptopenjdk user the last centos6_build_image (a year ago) was pushed by @gdams own user ID by the look of it, so I wonde if that's why adoptopenjdk can't override it.
Although that wouldn't explain any block on creating the ubuntu2004 one, unless wed prevented pushing of newly named images.
George, do you know the history of what's happened to this repository that might be causing these failures? And do you have an account that can log into it? Doesn't look like adoptopenjdk (as an organisation account) can log into the dockerhub web interface.
@sxa Do you think removing the latest centos6_build_image image in dockerhub and then re-pushing an image would help? The dockerhub credentials I have can login to docker in the cli but not into hub.docker.com else I would have tested this theory
@sxa Do you think removing the latest centos6_build_image image in dockerhub and then re-pushing an image would help? The dockerhub credentials I have can login to docker in the cli but not into hub.docker.com else I would have tested this theory
Adding to this, it might be possible to view a permissions config for each repo in the adoptopenjdk dockerhub. Perhaps something changed a year ago without notifying us? Again I cannot login to docker hub using the cli credentials so if someone who can login to docker could check?
The username and password worked for me this time, that's strange. I'll do some digging
Noting that while centos7_build_image was pushed by the adoptopenjdk user the last centos6_build_image (a year ago) was pushed by @gdams own user ID by the look of it, so I wonde if that's why adoptopenjdk can't override it.
The adoptopenjdkuser account, used for automated pushing, is of the automated team not the owners team. I assume members of the owners team have greater permissions, @gdams's account is of the owners team so @sxa might be right in thinking the adoptopenjdkuser user might not be able to override an image published by a user of the owners account
@sxa Do you think removing the latest centos6_build_image image in dockerhub and then re-pushing an image would help? The dockerhub credentials I have can login to docker in the cli but not into hub.docker.com else I would have tested this theory
The adoptopenjdk user looks like it does not have permissions to delete an image/tag. I think only owners can do that, so @gdams @tellison and @karianna .
I recommend one of the owners delete the latest tag of centos6_build_image and then we can see if the adoptopenjdkuser account is able to push a new centos6 image
Though we should first confirm if an owner account can push an image to centos6_build_image before deleting the latest image
@sxa Do you think removing the latest centos6_build_image image in dockerhub and then re-pushing an image would help?
The issue with that is because we are storing SHAs of docker files in the SBoMs, deleting the SHAs would invalidate the values in there, so we should avoid deleting existing ones if at all possible.
@tellison added a permission to the centos6 repo to allow the adoptopenjdkuser user to push images to the centos6 repo. It seems that the automated user needs a specific rule to allow it to push to the repo. Tim also created the adoptopenjdk/ubuntu2004_build_image:linux-riscv64 repo and then gave the automated user write permissions to solve the same error we were seeing with this image
WIth this solved I think its good to put the centos6 build back into https://ci.adoptium.net/job/centos7_docker_image_updater/ https://github.com/adoptium/infrastructure/pull/3296
Closing this now that https://github.com/adoptium/infrastructure/pull/3296 is merged and the centos6 part of https://ci.adoptium.net/job/centos7_docker_image_updater/ passes