ceph-build
ceph-build copied to clipboard
ceph
ceph-dev-build,ceph-dev-new-build: build c9 containers
is it safe to run this same ceph-container logic from multiple jobs? i was scanning through the log of a centos 8 job to see what it did, and noticed one line that looked suspicious:
+ docker push quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:34242b0fa30a22692c0ceab595cc3720eb89690c
(edit: this was from https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/78284/consoleText)
this is pushing something where the commit id 34242b0fa30a22692c0ceab595cc3720eb89690c is the only identitifier. if we run this from both centos 8 and 9 jobs, won't they conflict here?
we'll eventually turn off the centos 8 jobs for main and squid, but they'll both continue to run for quincy and reef branches
maybe quincy and reef should continue to build containers for just centos 8, while later branches build for 9 only? that way we'd only ever push one container image to quay per commit hash
That's a good point; prior to this there was only one release. Do we need to build c8 containers at all anymore?
On second thought, I think this needs to be a clean break. Changed to "unconditionally build only c9". Subject still under discussion.
we should only be changing the container's base distro for main/squid branches. the existing quincy/reef releases should remain based on c8
we should only be changing the container's base distro for main/squid branches. the existing quincy/reef releases should remain based on c8
Agreed, we want to minimize changes to the stable releases - upgrading the container distro is a more major change since it affects so many libraries.
maybe quincy and reef should continue to build containers for just centos 8, while later branches build for 9 only? that way we'd only ever push one container image to quay per commit hash
I like this idea, seems like it would obviate tagging and test changes to support more than one base container per branch
we could also change the format of the tags to just include a distroversion. I advocated for that when we started.
Currently investigating how we might test c9 containers. I'm thinking of a phased rollout for at least the newer releases (although I don't see why holding off on earlier releases should be necessary; do we not yet build anything before squid for non-container installs on c9?) but anyway, we could change the ceph dev builds to build and push the three tags for c8 and just the full tag for c9. I believe that cephadm can take an "image" config item to fully specify the image to test, but have yet to confirm that.
we could also change the format of the tags to just include a distroversion. I advocated for that when we started.
Currently investigating how we might test c9 containers. I'm thinking of a phased rollout for at least the newer releases (although I don't see why holding off on earlier releases should be necessary; do we not yet build anything before squid for non-container installs on c9?) but anyway, we could change the ceph dev builds to build and push the three tags for c8 and just the full tag for c9. I believe that cephadm can take an "image" config item to fully specify the image to test, but have yet to confirm that.
I think most of the time we just build the image name to be pulled by just combining quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph or quay.ceph.io/ceph-ci/ceph and the build sha. There are options to specify an image, but I'm not sure many (or any) tests actually do that (nothing in the orch/cephadm suite does). So changing the image tags in some way is likely to make things temporarily break there until the way the image name we pull is constructed is also updated. If the centos 9 images are going to use the same sha when building the image name we push, just getting main to build the container during the centos 9 builds rather than centos 8 would work immediately, at least in terms of pulling (no guarantee on if the images themselves will actually work). That might be the easier way to try and test them though, given things would break anyway if we changed the tag structure.
I'm thinking that targeted test runs with the proper config to specify the image:tag would serve as proof that the c9 containers work, and then if we wish we can decide to use the existing (2/3 non-distrover-bearing) tags for c9, or modify the consumers, or whatever. But for the first "are the containers even viable" test I'm hoping specifying the image name in the teuthology config is doable. Testing now.
(yes, when running teuthology tests you can override the image name used, at least by the cephadm task, which I think is the core of all our containerized tests, with an override of the toplevel "image" item to the full path of the container image you want.)
https://github.com/ceph/ceph-container/pull/2206 will fix the tag collision issue; once that merges, we can merge this
https://github.com/ceph/ceph-container/pull/2206 has been merged. @dmick merge this one as you see fit.
