Can't build container images using mock

[davidcassany]

Short description of the problem

Container toolchain is not fully functional inside a mock environment. More specific I can't manage to run buildah commit command without errors inside a mock environment.

Output of rpm -q mock

host os: Fedora Linux 36 mock: mock-3.0-1.fc36.noarch

Steps to reproduce issue

As root

1. mock -r fedora-36-x86_64 --init
2. mock -r fedora-36-x86_64 --install buildah
3. mock -r fedora-36-x86_64 --isolation simple --enable-network --shell

Inisde the mock env:

# just dummy file to add something inside the container
dd if=/dev/random of=data bs=1024 count=10240

# Create an empty container from scratch
buildah from --name test-container scratch

# Mount the created container
mnt=$(buildah mount test-container)

# Add dummy data in it
cp data ${mnt}

# Umount the container
buildah umount test-container

# Add some metadata to the container
buildah config --author="David Cassany <[email protected]>" test-container

# Commit the container into a container image, here it fails
buildah commit --rm --format oci test-container test-image:latest

The error message is:

Getting image source signatures
Copying blob ae378299c037 done  
error committing container "test-container" to "test-image:latest": error copying layers and metadata for container "2c48e86798dc1dc86e9e20752793cb3986f3814b4fcf3b9f49fe01ff7a54eb28": writing blob: adding layer with blob "sha256:ae378299c037f42b2190ccdc9201bff4e341ffe42d187ca44c63e3ba3b2c1a6f": Error processing tar file(exit status 1): remount /, flags: 0x44000: invalid argument

Feels like the mock environment is not properly set to handle the container storage, I tried creating an /etc/containers/storage.confg file and apply different drivers but I always got the same exact result.

Feels like this could be something related about how the chroot is set and configured...

Any additional notes

This issues has been created as a result of attempting to integrate mock and KIWI together. In fact this is the last rock block we have to consider KIWI functional within mock. Issue tracker here https://github.com/OSInside/kiwi/issues/2020

[xsuchy]

For the record - it works (for me) when I run it with systemd-nspawn:

mock -r fedora-36-x86_64 --isolation nspawn --enable-network --shell

[Conan-Kudo]

cc: @ignatenkobrain

[Conan-Kudo]

cc: @tkopecek

[praiskup]

This just worked for me even with --isolation simple. Can you please re-try if it still happens?

$ mock -r fedora-36-x86_64 --isolation simple --enable-network --shell
INFO: mock.py version 3.1 starting (python version = 3.10.7, NVR = mock-3.1-1.git.3448.b1ebac9.fc36)...
Start(bootstrap): init plugins
INFO: tmpfs initialized
INFO: selinux enabled
INFO: compress_logs: initialized
Finish(bootstrap): init plugins
Start: init plugins
INFO: tmpfs initialized
INFO: selinux enabled
INFO: compress_logs: initialized
Finish: init plugins
INFO: Signal handler active
Start: run
Start(bootstrap): chroot init
INFO: mounting tmpfs at /var/lib/mock/fedora-36-x86_64-bootstrap/root.
INFO: reusing tmpfs at /var/lib/mock/fedora-36-x86_64-bootstrap/root.
INFO: calling preinit hooks
INFO: enabled root cache
INFO: enabled package manager cache
Start(bootstrap): cleaning package manager metadata
Finish(bootstrap): cleaning package manager metadata
INFO: enabled HW Info plugin
Finish(bootstrap): chroot init
Start: chroot init
INFO: mounting tmpfs at /var/lib/mock/fedora-36-x86_64/root.
INFO: reusing tmpfs at /var/lib/mock/fedora-36-x86_64/root.
INFO: calling preinit hooks
INFO: enabled root cache
INFO: enabled package manager cache
Start: cleaning package manager metadata
Finish: cleaning package manager metadata
INFO: enabled HW Info plugin
Finish: chroot init
Start: shell
<mock-chroot> sh-5.1# dd if=/dev/random of=data bs=1024 count=10240
10240+0 records in
10240+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0433138 s, 242 MB/s
<mock-chroot> sh-5.1# buildah from --name test-container scratch
test-container
<mock-chroot> sh-5.1# mnt=$(buildah mount test-container)
<mock-chroot> sh-5.1# echo $mnt
/var/lib/containers/storage/overlay/6470df136b56a80d0e6d7c2513d6ff5ca54826807d0247f3bcb910bf26885938/merged
<mock-chroot> sh-5.1# cp data ${mnt}
<mock-chroot> sh-5.1# buildah umount test-container
e48da7e82ac131544c0377b811cfda6aa3430b50e3a06d33d08e61976a33d7dd
<mock-chroot> sh-5.1# buildah config --author="David Cassany <[email protected]>" test-container
<mock-chroot> sh-5.1# buildah commit --rm --format oci test-container test-image:latest
Getting image source signatures
Copying blob 5a8bc159bbaf done  
Copying config 9bb56f6a92 done  
Writing manifest to image destination
Storing signatures
9bb56f6a923714a24f8c7bcdbcbdc2b018fcbd0099bcb3ddd5726471ba5d5f97
<mock-chroot> sh-5.1# echo $?
0

[xsuchy]

Ping @davidcassany can you help us to provide the reproducer? Otherwise, I will close this report.

[Conan-Kudo]

I believe @ignatenkobrain can provide a reproducer, but on the KIWI side, we did recently make an adjustment which might help: https://github.com/OSInside/kiwi/pull/2195

I don't know if that fully resolves this issue, but as @ignatenkobrain was the original one who discovered the problem, he can validate whether it's fixed now.

[ignatenkobrain]

I just tried it on latest greatest CentOS Stream 9 with latest mock/kiwi from EPEL and it still does not work…

Mock indeed does bind-mount:

DEBUG util.py:535:  Using nspawn with args ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.zoubxl9y:/etc/resolv.conf', '--bind=/dev/btrfs-control', '--bind=/dev/mapper/control', '--bind=/dev/loop-control', '--bind=/dev/loop0', '--bind=/dev/loop1', '--bind=/dev/loop2', '--bind=/dev/loop3', '--bind=/dev/loop4', '--bind=/dev/loop5', '--bind=/dev/loop6', '--bind=/dev/loop7', '--bind=/dev/loop8', '--bind=/dev/loop9', '--bind=/dev/loop10', '--bind=/dev/loop11']

But kiwi fails with:

DEBUG util.py:445:  [ INFO    ]: 15:16:00 | Creating raw disk image /builddir/result/image/gdc-c9s-Cloud.x86_64-0.0.2.raw
DEBUG util.py:445:  [ INFO    ]: 15:16:01 | --> using all_freeMB for the root(rw) partition if present
DEBUG util.py:445:  [ INFO    ]: 15:16:01 | --> creating root partition [with 0 clone(s)]
DEBUG util.py:445:  [ INFO    ]: 15:16:01 | --> setting active flag to primary boot partition
DEBUG util.py:445:  [ INFO    ]: 15:16:01 | --> setting start sector to: 2048
DEBUG util.py:443:  [ ERROR   ]: 15:16:01 | KiwiCommandError: partx: stderr: partx: /dev/loop1: error adding partition 1
DEBUG util.py:443:  , stdout: (no output on stdout)
DEBUG util.py:445:  [ INFO    ]: 15:16:01 | Cleaning up LoopDevice instance

[ignatenkobrain]

Just realized that this specific ticket is talking about OCI stuff which indeed works nowadays.

[praiskup]

So can anyone identify the problem, what can we do about this in the Mock project?

[praiskup]

Closing as inactive. Feel free to reopen.

[Conan-Kudo]

I think we need to kick this to systemd-nspawn.