skopeo
skopeo copied to clipboard
containers
[skip-ci] Packit/TMT: Run gating tests
This commit will run Skopeo's gating tests using Packit and TMT.
If all tests can be run via Packit + TMT, it should be a cheaper alternative to Cirrus (except the Debian part).
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@lsm5 are you still working on this?
(If so, this is not urgent from my POV, no need to push it to the front of your queue — I’d just like to close the PR if there is absolutely no chance this work will continue.)
@mtrmac I do plan to work on this. But may take a while. I have bookmarked it for myself so I don't lose track. So please feel free to close it.
Thanks, if it is still relevant, let’s keep it open. It might help, or inspire, other contributors.
@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running test-system-local ? Asking mainly RE: packit + tmt tests.
Forgive me please, I don't understand the question. It's been a while since I've looked at skopeo gating tests, but my recollection is that they run on a plain system, via dnf install skopeo-tests. What's the context for "in a container"?
@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running
test-system-local?
IIRC the system tests run a registry server (container). So… running -local would be much simpler in that there would be no need for nested Podman; OTOH the tests would be much less isolated from the surrounding environment, which might be a difficulty.
I’m not immediately sure how much custom tooling exists in the CI container, and would need to be reproduced; https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh is relevant to that, but IIRC almost all of that exists for integration, not system, tests.
So, unfortunately, the issue we are hitting is a bug in Packit:
https://github.com/packit/packit-service/issues/2028
It was not considered a bug before, as people were not hitting it and we did not know it had this consequence. They are working on a fix.
So, unfortunately, the issue we are hitting is a bug in Packit:
It was not considered a bug before, as people were not hitting it and we did not know it had this consequence. They are working on a fix.
Thanks @thrix . I'll disable the rhel tests for now.
@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running
test-system-local?IIRC the system tests run a registry server (container). So… running
-localwould be much simpler in that there would be no need for nested Podman; OTOH the tests would be much less isolated from the surrounding environment, which might be a difficulty.I’m not immediately sure how much custom tooling exists in the CI container, and would need to be reproduced; https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh is relevant to that, but IIRC almost all of that exists for integration, not system, tests.
ack, the test-system target seems to spawn some container too. Anyway, I'll try to mimic the cirrus setup in tmt. Thanks
![packit-as-a-service[bot] avatar](https://avatars.githubusercontent.com/in/29076?v=4 "Published by a pub.dev verified publisher"){kind=small}
@mtrmac PTAL. Only added x86_64 integration, validate and unit tests for now.
If -local targets in CI tests are acceptable, that'd save us the pain of quay.io/libpod/skopeo_cidev image maintenance. Else, I'll look at making that image multiarch.
Hmm, and I guess running the tests in the skopeo_cidev container renders the multiple testing-farm- distro tests pointless.
If
-localtargets in CI tests are acceptable, that'd save us the pain of quay.io/libpod/skopeo_cidev image maintenance. Else, I'll look at making that image multiarch.
From an extremely brief look at the Packit / TMT documentation, the tests seem to run in an isolated VM (chroot? container?) and install RPM test subjects into that. Running an extra level of isolation in a container seems unnecessary in that case. (Also, to my knowledge no-one has been using the $CONTAINER_RUN targets for some time, they might have been broken for months without anyone noticing.) So, using the -local targets does seem better to me.
Also, testing the binary, if it comes from a RPM, in its expected distribution, seems clearly more valuable than running it in some other frozen / differently-built container.
OTOH, at least one reason why the container exists, and is regularly rebuilt, is so that individual Skopeo CI runs don’t need to install and compile all of those tools, which rarely change, on every single CI RUN. Hence, in Cirrus, runner.sh uses the -local targets but fetches pre-built binaries from that container.
If the Packit tests are ever intended to be merge-blocking, something like that to shorten the CI time would be very valuable.
And, also, I’d prefer not to maintain the test tool setup code ( https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh and make tools ) in two duplicated copies, if that could be helped.
@mtrmac thanks a lot for all the comments. Setting back to draft while I work through them.