plugins: Add bootc provision plugin

[ckyrouac]

This creates a new provision plugin that is built on top of the existing TestCloud (virtual) plugin. It adds new parameters to pass a Containerfile or container image. The plugin will then build a container image (if necessary) then build a bootc disk image from the container image using bootc image builder. Currently, bootc requires podman to be run as root when building a disk image. This is typically handled by running a podman machine as root.

An additional parameter "add-deps" toggles building a derived container image with the tmt test requirements.

Pull Request Checklist

• [X] implement the feature
• [ ] write the documentation
• [ ] extend the test coverage
• [ ] update the specification
• [ ] adjust plugin docstring
• [ ] modify the json schema
• [ ] mention the version
• [ ] include a release note

---

This is a work in progress. I'm opening this PR early to get feedback on the high level design. I will add tests, docs, etc. after we solidify the higher level design.

If you want to try running the code here is an example fmf plan:

provision:
  how: bootc
  containerimage: quay.io/centos-bootc/centos-bootc:stream9
  disk: 20
summary: Testing bootc plugin
execute:
  how: tmt
  script: |
    echo "ok"

[ckyrouac]

@cgwalters fyi

[happz]

• LGTM, I like the reuse of existing plugins, that's very nice.
• Would it also make sense to support the container plugin? IIUIC, the image produced by podman can be run both with podman run, or, converted to qcow2, as a VM. I could imagine both ways being available to use, maybe through some runtime-plugin: container|virtual switch. Is it a bad idea?
• I will have more low-level comments related to the actual implementation, but I won't bother you now till you receive the high-level review. Just ping when you're ready for the boring stuff :)

[happz]

I'm still trying to speedrun reading/learning all things bootc (and tmt provisioning plugins), but fwiw, looks cool to me.

@happz about the container plugin support - Fedora docs says:

for fully-fledged tests it is not recommended to run a bootable container via, for instance, podman-run. One reason among others is that the filesystem is writable when being executed as an OCI container while most of the filesystem is mounted read-only on a deployed bootc system. That means the running container behaves differently than a deployed system. Yet, if you desire to run some quick tests it is recommended to run the container in detached mode.

From what I understand, podman-bootc could be pretty cool to use, once available.

I agree, it's not a perfect 1:1 substitution, but, exactly: for quick tests or basic test development, it may give me results faster than VM. I for one work on binutils and C/C++ toolchain in general, and my area of focus is fairly simple - compile this, run objdump on that, grep for expected section names, this kind of stuff. Learning about a typo in my reproducer quickly is very valuable, and once I'm done, I can always use full VM. I do it today already: I develop tests with container, then I switch to beaker or 1minutetip to get a more real environment before committing the new test to git. So, podman-run, for my trivial component, would be very welcome, even with caveats like the one you mentioned :)

[martinhoyer]

I agree, it's not a perfect 1:1 substitution, but, exactly: for quick tests or basic test development, it may give me results faster than VM. I for one work on binutils and C/C++ toolchain in general, and my area of focus is fairly simple - compile this, run objdump on that, grep for expected section names, this kind of stuff. Learning about a typo in my reproducer quickly is very valuable, and once I'm done, I can always use full VM. I do it today already: I develop tests with container, then I switch to beaker or 1minutetip to get a more real environment before committing the new test to git. So, podman-run, for my trivial component, would be very welcome, even with caveats like the one you mentioned :)

Thank you for the insight in your development process. I hope I can see it in more detail one day.

[ckyrouac]

Would it also make sense to support the container plugin? IIUIC, the image produced by podman can be run both with podman run, or, converted to qcow2, as a VM. I could imagine both ways being available to use, maybe through some runtime-plugin: container|virtual switch. Is it a bad idea?

I think the existing container plugin will handle this case without any additional code. The bootc image is just another container that can be run like a typical image.

[ckyrouac]

@happz This is ready for a review. I added some docs, tests, and code to cleanup the container images.

[ckyrouac]

@happz thanks for the review! I believe I addressed all your suggestions.

[martinhoyer]

@happz let's rebase and run the pipeline here?

[thrix]

We agreed to focus on getting this in 1.38, the due date for finishing is 24th October.

[ckyrouac]

thanks for looking at this and planning for 1.38. Is there anything I can do to help move it along?

[thrix]

thanks for looking at this and planning for 1.38. Is there anything I can do to help move it along?

@ckyrouac not really, the team is overhelmed with reviews.

I am getting validation errors, I assume the jsonschema needs an update for the new provision plugin:

❯ pwd
/var/home/mvadkert/git/github.com/teemtee/tmt/tests/provision/bootc/data

❯ tmt lint
    warn: /containerfile_includes_deps:provision - {'how': 'bootc', 'add-deps': False, 'containerfile': '$TMT_BOOTC_CONTAINERFILE_RUNDIR/includes_deps.containerfile', 'containerfile-workdir': '.', 'disk': 20} is not valid under any of the given schemas
    warn: /containerfile_needs_deps:provision - {'how': 'bootc', 'add-deps': True, 'containerfile': '$TMT_BOOTC_CONTAINERFILE_RUNDIR/needs_deps.containerfile', 'containerfile-workdir': '.', 'disk': 20} is not valid under any of the given schemas
    warn: /image_includes_deps:provision - {'how': 'bootc', 'add-deps': False, 'containerimage': 'localhost/tmt-bootc-includes-deps', 'disk': 20} is not valid under any of the given schemas
    warn: /image_needs_deps:provision - {'how': 'bootc', 'add-deps': True, 'containerimage': 'localhost/tmt-bootc-needs-deps', 'disk': 20} is not valid under any of the given schemas
/containerfile_includes_deps
warn C000 key "add-deps" not recognized by schema /schemas/provision/container
warn C000 key "containerfile" not recognized by schema /schemas/provision/container
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/container
warn C000 key "disk" not recognized by schema /schemas/provision/container
warn C000 value of "how" is not "container"
warn C000 key "add-deps" not recognized by schema /schemas/provision/mrack
warn C000 key "containerfile" not recognized by schema /schemas/provision/mrack
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/mrack
warn C000 key "disk" not recognized by schema /schemas/provision/mrack
warn C000 value of "how" is not "beaker"
warn C000 key "add-deps" not recognized by schema /schemas/provision/virtual
warn C000 key "containerfile" not recognized by schema /schemas/provision/virtual
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/virtual
warn C000 value of "how" is not "virtual"
warn C000 key "add-deps" not recognized by schema /schemas/provision/artemis
warn C000 key "containerfile" not recognized by schema /schemas/provision/artemis
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/artemis
warn C000 key "disk" not recognized by schema /schemas/provision/artemis
warn C000 value of "how" is not "artemis"
warn C000 key "add-deps" not recognized by schema /schemas/provision/minute
warn C000 key "containerfile" not recognized by schema /schemas/provision/minute
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/minute
warn C000 key "disk" not recognized by schema /schemas/provision/minute
warn C000 value of "how" is not "minute"
warn C000 key "add-deps" not recognized by schema /schemas/provision/connect
warn C000 key "containerfile" not recognized by schema /schemas/provision/connect
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/connect
warn C000 key "disk" not recognized by schema /schemas/provision/connect
warn C000 value of "how" is not "connect"
warn C000 key "add-deps" not recognized by schema /schemas/provision/local
warn C000 key "containerfile" not recognized by schema /schemas/provision/local
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/local
warn C000 key "disk" not recognized by schema /schemas/provision/local
warn C000 value of "how" is not "local"
warn C000 fmf node failed schema validation
warn C001 summary key is missing
pass P001 correct keys are used
pass P002 execute step defined with "how"
pass P003 execute step methods are all known
pass P004 discover step methods are all known
skip P005 no remote fmf ids defined
pass P006 phases have unique names
pass P007 execute phase 'default-0' does not require specific guest

/containerfile_needs_deps
warn C000 key "add-deps" not recognized by schema /schemas/provision/container
warn C000 key "containerfile" not recognized by schema /schemas/provision/container
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/container
warn C000 key "disk" not recognized by schema /schemas/provision/container
warn C000 value of "how" is not "container"
warn C000 key "add-deps" not recognized by schema /schemas/provision/mrack
warn C000 key "containerfile" not recognized by schema /schemas/provision/mrack
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/mrack
warn C000 key "disk" not recognized by schema /schemas/provision/mrack
warn C000 value of "how" is not "beaker"
warn C000 key "add-deps" not recognized by schema /schemas/provision/virtual
warn C000 key "containerfile" not recognized by schema /schemas/provision/virtual
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/virtual
warn C000 value of "how" is not "virtual"
warn C000 key "add-deps" not recognized by schema /schemas/provision/artemis
warn C000 key "containerfile" not recognized by schema /schemas/provision/artemis
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/artemis
warn C000 key "disk" not recognized by schema /schemas/provision/artemis
warn C000 value of "how" is not "artemis"
warn C000 key "add-deps" not recognized by schema /schemas/provision/minute
warn C000 key "containerfile" not recognized by schema /schemas/provision/minute
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/minute
warn C000 key "disk" not recognized by schema /schemas/provision/minute
warn C000 value of "how" is not "minute"
warn C000 key "add-deps" not recognized by schema /schemas/provision/connect
warn C000 key "containerfile" not recognized by schema /schemas/provision/connect
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/connect
warn C000 key "disk" not recognized by schema /schemas/provision/connect
warn C000 value of "how" is not "connect"
warn C000 key "add-deps" not recognized by schema /schemas/provision/local
warn C000 key "containerfile" not recognized by schema /schemas/provision/local
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/local
warn C000 key "disk" not recognized by schema /schemas/provision/local
warn C000 value of "how" is not "local"
warn C000 fmf node failed schema validation
warn C001 summary key is missing
pass P001 correct keys are used
pass P002 execute step defined with "how"
pass P003 execute step methods are all known
pass P004 discover step methods are all known
skip P005 no remote fmf ids defined
pass P006 phases have unique names
pass P007 execute phase 'default-0' does not require specific guest

/image_includes_deps
warn C000 key "add-deps" not recognized by schema /schemas/provision/container
warn C000 key "containerimage" not recognized by schema /schemas/provision/container
warn C000 key "disk" not recognized by schema /schemas/provision/container
warn C000 value of "how" is not "container"
warn C000 key "add-deps" not recognized by schema /schemas/provision/mrack
warn C000 key "containerimage" not recognized by schema /schemas/provision/mrack
warn C000 key "disk" not recognized by schema /schemas/provision/mrack
warn C000 value of "how" is not "beaker"
warn C000 key "add-deps" not recognized by schema /schemas/provision/virtual
warn C000 key "containerimage" not recognized by schema /schemas/provision/virtual
warn C000 value of "how" is not "virtual"
warn C000 key "add-deps" not recognized by schema /schemas/provision/artemis
warn C000 key "containerimage" not recognized by schema /schemas/provision/artemis
warn C000 key "disk" not recognized by schema /schemas/provision/artemis
warn C000 value of "how" is not "artemis"
warn C000 key "add-deps" not recognized by schema /schemas/provision/minute
warn C000 key "containerimage" not recognized by schema /schemas/provision/minute
warn C000 key "disk" not recognized by schema /schemas/provision/minute
warn C000 value of "how" is not "minute"
warn C000 key "add-deps" not recognized by schema /schemas/provision/connect
warn C000 key "containerimage" not recognized by schema /schemas/provision/connect
warn C000 key "disk" not recognized by schema /schemas/provision/connect
warn C000 value of "how" is not "connect"
warn C000 key "add-deps" not recognized by schema /schemas/provision/local
warn C000 key "containerimage" not recognized by schema /schemas/provision/local
warn C000 key "disk" not recognized by schema /schemas/provision/local
warn C000 value of "how" is not "local"
warn C000 fmf node failed schema validation
warn C001 summary key is missing
pass P001 correct keys are used
pass P002 execute step defined with "how"
pass P003 execute step methods are all known
pass P004 discover step methods are all known
skip P005 no remote fmf ids defined
pass P006 phases have unique names
pass P007 execute phase 'default-0' does not require specific guest

/image_needs_deps
warn C000 key "add-deps" not recognized by schema /schemas/provision/container
warn C000 key "containerimage" not recognized by schema /schemas/provision/container
warn C000 key "disk" not recognized by schema /schemas/provision/container
warn C000 value of "how" is not "container"
warn C000 key "add-deps" not recognized by schema /schemas/provision/mrack
warn C000 key "containerimage" not recognized by schema /schemas/provision/mrack
warn C000 key "disk" not recognized by schema /schemas/provision/mrack
warn C000 value of "how" is not "beaker"
warn C000 key "add-deps" not recognized by schema /schemas/provision/virtual
warn C000 key "containerimage" not recognized by schema /schemas/provision/virtual
warn C000 value of "how" is not "virtual"
warn C000 key "add-deps" not recognized by schema /schemas/provision/artemis
warn C000 key "containerimage" not recognized by schema /schemas/provision/artemis
warn C000 key "disk" not recognized by schema /schemas/provision/artemis
warn C000 value of "how" is not "artemis"
warn C000 key "add-deps" not recognized by schema /schemas/provision/minute
warn C000 key "containerimage" not recognized by schema /schemas/provision/minute
warn C000 key "disk" not recognized by schema /schemas/provision/minute
warn C000 value of "how" is not "minute"
warn C000 key "add-deps" not recognized by schema /schemas/provision/connect
warn C000 key "containerimage" not recognized by schema /schemas/provision/connect
warn C000 key "disk" not recognized by schema /schemas/provision/connect
warn C000 value of "how" is not "connect"
warn C000 key "add-deps" not recognized by schema /schemas/provision/local
warn C000 key "containerimage" not recognized by schema /schemas/provision/local
warn C000 key "disk" not recognized by schema /schemas/provision/local
warn C000 value of "how" is not "local"
warn C000 fmf node failed schema validation
warn C001 summary key is missing
pass P001 correct keys are used
pass P002 execute step defined with "how"
pass P003 execute step methods are all known
pass P004 discover step methods are all known
skip P005 no remote fmf ids defined
pass P006 phases have unique names
pass P007 execute phase 'default-0' does not require specific guest

    warn: /containerfile_includes_deps:provision - {'how': 'bootc', 'add-deps': False, 'containerfile': '$TMT_BOOTC_CONTAINERFILE_RUNDIR/includes_deps.containerfile', 'containerfile-workdir': '.', 'disk': 20, 'name': 'default-0'} is not valid under any of the given schemas
    warn: /containerfile_needs_deps:provision - {'how': 'bootc', 'add-deps': True, 'containerfile': '$TMT_BOOTC_CONTAINERFILE_RUNDIR/needs_deps.containerfile', 'containerfile-workdir': '.', 'disk': 20, 'name': 'default-0'} is not valid under any of the given schemas
    warn: /image_includes_deps:provision - {'how': 'bootc', 'add-deps': False, 'containerimage': 'localhost/tmt-bootc-includes-deps', 'disk': 20, 'name': 'default-0'} is not valid under any of the given schemas
    warn: /image_needs_deps:provision - {'how': 'bootc', 'add-deps': True, 'containerimage': 'localhost/tmt-bootc-needs-deps', 'disk': 20, 'name': 'default-0'} is not valid under any of the given schemas
Lint checks on all
pass G001 no duplicate ids detected

[thrix]

/packit build

[thrix]

I am getting validation errors, I assume the jsonschema needs an update for the new provision plugin:

❯ pwd
/var/home/mvadkert/git/github.com/teemtee/tmt/tests/provision/bootc/data

❯ tmt lint
    warn: /containerfile_includes_deps:provision - {'how': 'bootc', 'add-deps': False, 'containerfile': '$TMT_BOOTC_CONTAINERFILE_RUNDIR/includes_deps.containerfile', 'containerfile-workdir': '.', 'disk': 20} is not valid under any of the given schemas
    warn: /containerfile_needs_deps:provision - {'how': 'bootc', 'add-deps': True, 'containerfile': '$TMT_BOOTC_CONTAINERFILE_RUNDIR/needs_deps.containerfile', 'containerfile-workdir': '.', 'disk': 20} is not valid under any of the given schemas
    warn: /image_includes_deps:provision - {'how': 'bootc', 'add-deps': False, 'containerimage': 'localhost/tmt-bootc-includes-deps', 'disk': 20} is not valid under any of the given schemas
    warn: /image_needs_deps:provision - {'how': 'bootc', 'add-deps': True, 'containerimage': 'localhost/tmt-bootc-needs-deps', 'disk': 20} is not valid under any of the given schemas
/containerfile_includes_deps
warn C000 key "add-deps" not recognized by schema /schemas/provision/container
warn C000 key "containerfile" not recognized by schema /schemas/provision/container
warn C000 key "containerfile-workdir" not recognized by schema /schemas/provision/container
warn C000 key "disk" not recognized by schema /schemas/provision/container
warn C000 value of "how" is not "container"
... snip ....

@ckyrouac FTR the respective schemas are in https://github.com/teemtee/tmt/tree/bc64a2fd569c2d1179cb63bd50dbefff429d4f14/tmt/schemas/provision, you will need to add one for this new provision plugin

[happz]

Bumping to 1.39: we decided to shorten the 1.38 release cycle and ship it tomorrow to deliver two important features. 1.39 should be also shortened, taking just 2 weeks.

[thrix]

@ckyrouac Let us know if you would need any help and I promise I will be more timely on the reviews. Thanks for bringing this feature to tmt and sorry this took so long.

[ckyrouac]

@thrix no worries. I pushed all the requested changes in a few commits to make it easier to review. I can squash after review if you want.

[happz]

/packit build

[happz]

A couple of comments, mostly to improve readability (and prevent --cpu and --memory from sneaking back into tmt when I finally got rid of them in virtual :)

@ckyrouac please, install and enable pre-commit in your repo. It runs various linters on your code, and they do have something to say about it, according to a failed check here.

[ckyrouac]

force pushed updates addressing all the suggestions.

I had broken out the podman-machine cpu/mem parameters into environment variables with the idea that the podman-machine would be spun up once for multiple tests. With the current code, a new machine is created and destroyed for each test run. That's mostly because I'm not sure where/how to hook into the higher level run code. Although, the more I think about it, I guess allowing each test to specify the podman-machine cpu/mem requirements makes sense because some tests might require significantly more resources than other tests.

please, install and enable pre-commit in your repo

hmm, I have it installed but it didn't run on one of those commits for some reason.

[psss]

/packit build

[ckyrouac]

hmm, looking into the test failures

[ckyrouac]

not sure why the CI tests were failing. The tests are passing locally, in CI the setup was failing to create the /var/tmp/tmt/test-XXXX directory. I can't find the previous run results to get the exact error after I pushed the latest updates.

[psss]

One of the old jobs is here. There seems to be a problem with tmt init at the beginning:

:: [ 09:08:32 ] :: [  BEGIN   ] :: Running 'tmt init'
Path '/var/ARTIFACTS/work-bootcdy90p5cu/plans/provision/bootc/discover/default-0/tests/tests/provision/bootc' already has a parent fmf tree root '/var/ARTIFACTS/work-bootcdy90p5cu/plans/provision/bootc/discover/default-0/tests/.fmf'.
Do you really want to initialize a nested tree? [y/N]: Aborted!
:: [ 09:08:33 ] :: [   FAIL   ] :: Command 'tmt init' (Expected 0, got 1)

Using tmt init --force should help.

[psss]

/packit build

[ckyrouac]

ah, thanks for the link. The latest commit should fix it. Not sure if you need to trigger another test run or not.

[psss]

/packit build

[ckyrouac]

/packit build

[happz]

/packit build

[happz]

/packit build

[ckyrouac]

The tests are failing with Failed to connect socket to '/var/run/libvirt/virtqemud-sock': No such file or directory. Any ideas why?