bootc-image-builder
bootc-image-builder copied to clipboard
osbuild
cross-architecture broken with recent bootc
I cannot build any cross-arch (ARM to AMD64) images on my M1 MacBook. It fails on some interesting layer mounting issue:
Steps:
- Installed Podman Desktop
-
podman machine stop podman machine set --rootful podman machine start - The most minimal example I could break is this:
sudo podman run \ --rm \ -it \ --privileged \ --pull=newer \ --security-opt label=type:unconfined_t \ -v $(pwd)/output:/output \ --platform linux/amd64 \ quay.io/centos-bootc/bootc-image-builder:latest \ --type qcow2 \ --target-arch amd64 \ quay.io/fedora/fedora-bootc:40
which results in:
[...]
Copying blob 8f11ed82e8bb done |
[...]
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:3f4b29abb14575e3968cf042ba63d35deed74aca2d746f96b547e6ca5da7539e"/""/"sha256:b1bc5c0fb77df66e970553b0fac7730ac3440fc85e5c619fc01929a83b926a8e": unpacking failed (error: exit status 125; output: Error: unrecognized command `podman /`
Did you mean this?
cp
ps
rm
Try 'podman --help' for more information
)
2024/09/09 12:33:47 error: cannot build manifest: failed to pull container image: exit status 125, stderr:
Just pulling that image normally with podman pull works fine.
I also originally ran into something very similar when trying to use --local:
sudo podman pull quay.io/fedora/fedora-bootc:40
sudo podman run \
--rm \
-it \
--privileged \
--pull=newer \
--security-opt label=type:unconfined_t \
-v $(pwd)/output:/output \
-v /var/lib/containers/storage:/var/lib/containers/storage \
--platform linux/amd64 \
quay.io/centos-bootc/bootc-image-builder:latest \
--local \
--type qcow2 \
--target-arch amd64 \
quay.io/fedora/fedora-bootc:40
which results in:
Generating manifest manifest-qcow2.json
2024/09/09 12:42:09 error: cannot build manifest: running quay.io/fedora/fedora-bootc:40 container failed: exit status 126
stderr:
WARNING: image platform (linux/arm64) does not match the expected platform (linux/amd64)
time="2024-09-09T12:42:09Z" level=error msg="Unmounting /var/lib/containers/storage/overlay/19aa28066f9c3a9a39ee20638c390f9890f17fd0bf3760ca004bf5129ad7026e/merged: invalid argument"
Error: mounting storage for container 30b6258182819f6b1cb6616cf505973fa71da50b6511e8055b012d4fb614f5b6: creating overlay mount to 19aa28066f9c3a9a39ee20638c390f9890f17fd0bf3760ca004bf5129ad7026e/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/WG7RWIHG44KOACMBKSDCFNP4UF:/var/lib/containers/storage/overlay/l/5E2NKIQA2XCJ3T4AQKVZP2R3UG:/var/lib/containers/storage/overlay/l/CPULKYB5OGOZNY7PDH6YKTQGCX:/var/lib/containers/storage/overlay/l/SQB6BFAROQNDVC6IX3CUFWSXFS:/var/lib/containers/storage/overlay/l/NBXZ7UEQXEL442FJ7GT5ACECY7:/var/lib/containers/storage/overlay/l/LD5N23GP5BWBQ32J3PSDKGGMSL:/var/lib/containers/storage/overlay/l/LOGZXIKFLKLIUTUX25U6TSBY3V:/var/lib/containers/storage/overlay/l/DGF44NJRKCLXE56QG35BKISWJE:/var/lib/containers/storage/overlay/l/JZNGUSGVYCGJTSYJAS42FYFJFE:/var/lib/containers/storage/overlay/l/62YV6HD2WSN45ENJQTUPNAYQAH:/var/lib/containers/storage/overlay/l/RZGBK6XY3BMFT7UAIONE2HWFDK:/var/lib/containers/storage/overlay/l/CFJIH5DM3AVXZNIMZOPEF44NP6:/var/lib/containers/storage/overlay/l/QLSMDFUFLGNCP7KFMMO4AEBIIX:/var/lib/containers/storage/overlay/l/EUR5I2TKHKZ25ZCZBYRESFKSKI:/var/lib/containers/storage/overlay/l/LV5LC2O23JARNKOWIXI5QLT673:/var/lib/containers/storage/overlay/l/5IBKL3MJ2PTFQLRI2PGEGW7JXZ:/var/lib/containers/storage/overlay/l/YYZAWX4TZVE2GJ4BCUBWMFGFM3:/var/lib/containers/storage/overlay/l/2MIR75TEUU2QC423UZCCTYEJ7P:/var/lib/containers/storage/overlay/l/T75WTLZ3BHVB6P73O6IRRQCPTE:/var/lib/containers/storage/overlay/l/W4CCR23R4FOMHGHFUSQ4EJZ6Y7:/var/lib/containers/storage/overlay/l/KLBKYDRVROGMGWN6QOBSKAJGZE:/var/lib/containers/storage/overlay/l/RWUKGPXV4GABE72P27TJ2BDDZL:/var/lib/containers/storage/overlay/l/W6VXGFGHRI6JYHFNOVHFEEPQ4U:/var/lib/containers/storage/overlay/l/5YEV7IODOM5BTO6AVUNRSX7EHR:/var/lib/containers/storage/overlay/l/XDWEFVS3QEIWIGPT5TT4BAU7PE:/var/lib/containers/storage/overlay/l/RSC3IISHFHHRK5JDRNQAJ6DTG5:/var/lib/containers/storage/overlay/l/YKDO6Q645735TKL2ICPUSP4HM3:/var/lib/containers/storage/overlay/l/BP65ZK6C7MVOLTYDOALUDUBFYY:/var/lib/containers/storage/overlay/l/AIBGFSGZQHGEJS67VL5G5BMK4T:/var/lib/containers/storage/overlay/l/XHFFVJZYQNEZRMKELOKKCQ2AFH:/var/lib/containers/storage/overlay/l/BG7UI7MF4LBFRGQURYIJQC5PDC:/var/lib/containers/storage/overlay/l/DKEODBUCU2KEYWJPD6J4XJQVOS:/var/lib/containers/storage/overlay/l/5N7BWBOCZ7PVLJDQUZB6LB75TS:/var/lib/containers/storage/overlay/l/6Y4VQO33NEIRNZUQPY3CBVEBED:/var/lib/containers/storage/overlay/l/3ZS4MCW77LAYRC6WXPIDSJ2HEU:/var/lib/containers/storage/overlay/l/H6JROLXARBSXEE3CBCJEHQ5ZAL:/var/lib/containers/storage/overlay/l/OX3SNSJM2UQW7MORLJ234DC6BP:/var/lib/containers/storage/overlay/l/YFAOEEOC2VCIRYYKUO6QREOO7H:/var/lib/containers/storage/overlay/l/TOBA7H7IXQYNFBAZQCVVXI3MOS:/var/lib/containers/storage/overlay/l/YT5EBCLSWFUUCHNI7YFTZIG2VJ:/var/lib/containers/storage/overlay/l/KLBD7GSW7RSEOULCAL7LY4NHCU:/var/lib/containers/storage/overlay/l/YSIV6BPFJW4XK2UWP6QVTKFZGO:/var/lib/containers/storage/overlay/l/METUZVYK753ZKNBVTVMXXQG2JQ:/var/lib/containers/storage/overlay/l/U4ZILNEHWLJLBJZYEF2F7DZN6U:/var/lib/containers/storage/overlay/l/4XDP4SSO7EWOGGA7DJ5HN73IGH:/var/lib/containers/storage/overlay/l/OL7MO6HJEPEKT7UVTUB5RWX3CY:/var/lib/containers/storage/overlay/l/3EJCJAT32TX7XGQ26ADQJJDZ7A:/var/lib/containers/storage/overlay/l/5MZCBFMCYSCTF7V5GJ5Y4KSNQC:/var/lib/containers/storage/overlay/l/N4ZG3X6ZZLT6ND3CHWECEYSZUS:/var/lib/containers/storage/overlay/l/3JZI6TJVTPOE2I4YBW64SE6IPI:/var/lib/containers/storage/overlay/l/PD6AGFRORP4WYJ7JLK3MXXQMSL:/var/lib/containers/storage/overlay/l/SO4NGGOJLD4AU3MGZSL5TAZDF5:/var/lib/containers/storage/overlay/l/QDJJUIUWYCVI5BX3LTL6ZAIIWQ:/var/lib/containers/storage/overlay/l/VJULNCEKKJQYNCGQODNUDPFKEV:/var/lib/containers/storage/overlay/l/Q3WWGFSNKVLMTXCTZBBQVHDV7A:/var/lib/containers/storage/overlay/l/TLIBJ2OH5D2NHN27WLZHU7DKDQ:/var/lib/containers/storage/overlay/l/D3YNL72ZOVWIHEEXFB55KKQDN5:/var/lib/containers/storage/overlay/l/BJPJVJRD2QMK4VTWPS4ZEMG3FX:/var/lib/containers/storage/overlay/l/KPS7ETLLPXT7OGSXK4BFZF7ZFU:/var/lib/containers/storage/overlay/l/CBFTKHUR56TDHWRFATJMJ4ZSLK:/var/lib/containers/storage/overlay/l/KBMR3ARDG5XV6Z2UCWXCHQAV7J:/var/lib/containers/storage/overlay/l/G7C2JS2HTYL5KO3XK5462JTFFE:/var/lib/containers/storage/overlay/l/NA5WHIZIE5X5HZBSXTXUERGT5E:/var/lib/containers/storage/overlay/l/FRRRIYN2BTWTZOGY3FCLSBEP5R:/var/lib/containers/storage/overlay/l/5TDPBY7VZYX342JKRNLQVJL3NJ,upperdir=19aa28066f9c3a9a39ee20638c390f9890f17fd0bf3760ca004bf5129ad7026e/diff,workdir=19aa28066f9c3a9a39ee20638c390f9890f17fd0bf3760ca004bf5129ad7026e/work,nodev,metacopy=on,volatile": mountfrom re-exec output: Error: unrecognized command `podman /var/lib/containers/storage/overlay`
Try 'podman --help' for more information
: error: exit status 125
All of this works fine on a Linux AMD64 machine/VM. Would appreciate some help to make the workflow painless again :)
Maybe fallout from https://github.com/containers/bootc/pull/724/commits/0527ca96202633625f79dfe06277b96cfb522000 - we're doing a lot more in bootc's install path now? I haven't tested cross arch in a while
Just noticed, my second example obviously has an issue as I pulled the wrong image by not specifying --arch.
sudo podman pull --arch amd64 quay.io/fedora/fedora-bootc:40
but the result is still the same:
Generating manifest manifest-qcow2.json
2024/09/09 14:31:40 error: cannot build manifest: running quay.io/fedora/fedora-bootc:40 container failed: exit status 126
stderr:
time="2024-09-09T14:31:40Z" level=error msg="Unmounting /var/lib/containers/storage/overlay/28498a00edd3e0f6be8e497d6ea6b5160cd6b199bd426591353507a633e4765a/merged: invalid argument"
Error: mounting storage for container 8456b0c200e84d9910d8a473b74f48f76cc930f8be4c9f7c14a4780ea387c643: creating overlay mount to 28498a00edd3e0f6be8e497d6ea6b5160cd6b199bd426591353507a633e4765a/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/DBHJ3TOIS4EK5VDIIT66EDJ7O6:/var/lib/containers/storage/overlay/l/6FP6MWRSJCAD3D6K5D4V6AM5NU:/var/lib/containers/storage/overlay/l/MV2AEFEG2DTO2OEU5S6VNIOYDC:/var/lib/containers/storage/overlay/l/KIGRTROGJXDCC237LRIBDWDJ3I:/var/lib/containers/storage/overlay/l/PGZH34LG4ATCHQOZSZQCMZEWLX:/var/lib/containers/storage/overlay/l/P2MGABDPKNOIXLZFGPYE3TYP2C:/var/lib/containers/storage/overlay/l/SN4EZWQLXDK5H7OQ73ZUHDN2LN:/var/lib/containers/storage/overlay/l/3Q6T647UMUCPNHJOIP3OLOTSXI:/var/lib/containers/storage/overlay/l/DGFCSZOIZRTTGRCF3N2OGTIUKX:/var/lib/containers/storage/overlay/l/ZW7U3GCVFBXQ6FMQPFTBNZBL7L:/var/lib/containers/storage/overlay/l/LE3MCF2DYHIYRDFJVPVGD34CGC:/var/lib/containers/storage/overlay/l/W4H3VXE5L3OITEILTKUVDJXZUP:/var/lib/containers/storage/overlay/l/JAX4533QZKFYSZAICJIFE7FHGF:/var/lib/containers/storage/overlay/l/ZJTUIB5K4O7GHW2MRXZ64S4RUO:/var/lib/containers/storage/overlay/l/DKUU4XJYEN2NVTIOFQB7ENOPNU:/var/lib/containers/storage/overlay/l/CGXNU2UK6CL6ELUPYMX5INEPGR:/var/lib/containers/storage/overlay/l/YKXKPBN74OKNMXM5NSID72RYZV:/var/lib/containers/storage/overlay/l/GAAJ6JD5YD6ZB2TXBOLG3XRUZO:/var/lib/containers/storage/overlay/l/EV6TILQS32YQAEHTCMUJVFRPX2:/var/lib/containers/storage/overlay/l/5RACCUIZSRAPTNICFNHWLEJ4MP:/var/lib/containers/storage/overlay/l/2KF42HEV3GZRQGHZMKPDNS7UYT:/var/lib/containers/storage/overlay/l/6OFFOR4TW7MQOKLMYK7YRM4V3O:/var/lib/containers/storage/overlay/l/T22CH5HJM4FUAS2NXTQPQPD3E5:/var/lib/containers/storage/overlay/l/MBSN7U2M6XJVQA3E3FPBQTTXWI:/var/lib/containers/storage/overlay/l/Z5NSFYFDKV627PP4GBVQVP3ZFZ:/var/lib/containers/storage/overlay/l/VBO5ZVU6C5LPP4QWZQ4FGE4I2E:/var/lib/containers/storage/overlay/l/UICGV2FPCT2SIJU6ZPJXKRUJEX:/var/lib/containers/storage/overlay/l/PTWUO3PWPMHDTCQATMFMYPAMC2:/var/lib/containers/storage/overlay/l/LET4MOTZHEA2YEF6JJ66MVA3ZO:/var/lib/containers/storage/overlay/l/BWJA3MSENPM7NQR6EQ64SLBKH2:/var/lib/containers/storage/overlay/l/YK3FYRK7PJM3KTDNU7VRI3K7RZ:/var/lib/containers/storage/overlay/l/TPIRFOQ4MB2LA6X7U2H734KYPP:/var/lib/containers/storage/overlay/l/7TRND6RHHS7XGKFZ5IHHUO23X6:/var/lib/containers/storage/overlay/l/IVBX5GKJ5LBZ2TJ3GXMX7BGOZI:/var/lib/containers/storage/overlay/l/MVQV7AAGPNCFYVLZPOYLUZRCIC:/var/lib/containers/storage/overlay/l/TAJPG443OA6OUPYZE3MPXL2MLW:/var/lib/containers/storage/overlay/l/XLK2UPFFB3L2HDEOE5IHCOAC3D:/var/lib/containers/storage/overlay/l/MSCRM23E4WUT3FGBMAFPV5RRHL:/var/lib/containers/storage/overlay/l/JYGGKS6N3YQSEIALANVAJIMX4F:/var/lib/containers/storage/overlay/l/22AIXX7YMH5W23LMKDBLT4ZMWP:/var/lib/containers/storage/overlay/l/EHQDWMERWFV4M66RIZ3OYPECQD:/var/lib/containers/storage/overlay/l/KR7VKYRL4XHKKRIRRIX34HX57P:/var/lib/containers/storage/overlay/l/THERXJR2RDXTUUQPMHBBMCR76D:/var/lib/containers/storage/overlay/l/JOEYPB2TCZHPGGMXQUV57JOHR6:/var/lib/containers/storage/overlay/l/2NBJHZHSNVZS4DGZYQTRVCXRPB:/var/lib/containers/storage/overlay/l/DSJBKRQCB4PDFPPLMBPTOQUEZE:/var/lib/containers/storage/overlay/l/ZXPSDZWLKQZCKNI26ODPZUVRFN:/var/lib/containers/storage/overlay/l/ISB3GNAAB5P5WFAC6RFLUWGQMR:/var/lib/containers/storage/overlay/l/YTUFMH52KLXEE3FV5FVEP5PUVG:/var/lib/containers/storage/overlay/l/VIKJXVNR7DPRHPCQZXB6P5HY4X:/var/lib/containers/storage/overlay/l/MADKRI7YIBA5DKOJV56YZUYK3F:/var/lib/containers/storage/overlay/l/QL3K4RQZOMTDZEAZU7T4IGA6BS:/var/lib/containers/storage/overlay/l/NKTIGYQZJ2NPSOHUYFFRAXKGDN:/var/lib/containers/storage/overlay/l/ICZTCBX2SU6HKXAC3IMAF6H3LN:/var/lib/containers/storage/overlay/l/BXIEMJX2XLEYRST7Q46SQHC654:/var/lib/containers/storage/overlay/l/VDY5OPWGAVBGERBVMCWV2RGNXZ:/var/lib/containers/storage/overlay/l/ZP5NTB4JJO6HTDO4UWZA54C4VZ:/var/lib/containers/storage/overlay/l/WY2N4OGB6ULM63T5CBASO4BWYP:/var/lib/containers/storage/overlay/l/7GIN2ATCGPQFDCPNE3QIZJWIZZ:/var/lib/containers/storage/overlay/l/BTQH6SDALHKIPS6QWVZUJEX5R5:/var/lib/containers/storage/overlay/l/B3V4K2EN5QBV55THT4ALDUGCQE:/var/lib/containers/storage/overlay/l/5SRR54EVAH6BJL3OYYKJY5DIUE:/var/lib/containers/storage/overlay/l/YNV2O4FSGJRB3WSNGV6OQCNB3M:/var/lib/containers/storage/overlay/l/76SFTAL3I75HMFG4OK6TXVHUOE:/var/lib/containers/storage/overlay/l/MAGBO7PRFFCEUEQAFHWG6DAUCM,upperdir=28498a00edd3e0f6be8e497d6ea6b5160cd6b199bd426591353507a633e4765a/diff,workdir=28498a00edd3e0f6be8e497d6ea6b5160cd6b199bd426591353507a633e4765a/work,nodev,metacopy=on,volatile": mountfrom re-exec output: Error: unrecognized command `podman /var/lib/containers/storage/overlay`
Try 'podman --help' for more information
: error: exit status 125
Maybe fallout from containers/bootc@0527ca9 - we're doing a lot more in bootc's install path now? I haven't tested cross arch in a while
We had cross arch as part of our bootc-image-builder CI until about a week or so ago, then it started breaking because of an unimplemented openat2 syscall in qemu (which is fixed in https://www.mail-archive.com/[email protected]/msg1064233.html but not accepted upstream yet) so this is currently on hold on our side.
Bigger picture it's going to be a bit tricky for us to maintain the qemu-emulated path as it is today; in bootc we really want to use some sophisticated Linux kernel features.
Probably the most fruitful avenue to pursue will be moving some cross-arch logic into bootc itself. It may to start just look something like bootc install ... --target-arch and just overriding the image we fetch. We'd need to work through any other details but I don't think there's a whole lot there.
FTR, the qemu-user openat2 support is now merged upstream, we "just" need to backport it to the podman-machine image (which is f40 afaik(?)).
Looks like:
- qemu-user 9.2 is the version that will contain this feature as far as i can tell (its the one whose feature freeze date was after this got merged).
- fedora 41 currently has version 9.1 packaged in the repos (
sudo dnf --releasever=41 --forcearch=aarch64 list qemu-user) - ~~it seems like there used to be a build of this under f40 at https://packages.fedoraproject.org/pkgs/qemu/qemu-user/~~
Am gonna see if i can compile this from latest sources and see if itll work
Looks like:
* qemu-user 9.2 is the version that will contain this feature as far as i can tell (its the one whose [feature freeze date](https://wiki.qemu.org/Planning/9.2) was after this got merged). * fedora 41 currently has version 9.1 packaged in the repos (`sudo dnf --releasever=41 --forcearch=aarch64 list qemu-user`)
[..] The fix got backported via https://src.fedoraproject.org/rpms/qemu/pull-request/70 into f41, the backport for f40 would be a bit more involved though as qemu upstream changed a bit more.
we "just" need to backport it to the podman-machine image (which is f40 afaik(?)).
I tried modifying the containerfile to include this copr repo, but that didn't seem to change the error message - maybe i did something wrong
my changes are here: https://github.com/MoralCode/bootc-image-builder/tree/f40
we "just" need to backport it to the podman-machine image (which is f40 afaik(?)).
I tried modifying the containerfile to include this copr repo, but that didn't seem to change the error message - maybe i did something wrong
my changes are here: https://github.com/MoralCode/bootc-image-builder/tree/f40
The qemu-user binaries need to come from the system, so either the host or the machine inside podman machine. The namespace support for the underlying binfmt_misc is relatively new [0] we should probably look into using this, then we could potentially have qemu-user as part of the bib container. Maybe worth filing a separate issue that we should look into supporting this via https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs.binfmt_misc&id=ecddcab2d1b15fea782889237093bd069979c8c7 as it will benefit fedora/rhel10/centos10
There is also an extra complication when running under the mac, here rosetta2 needs to be disabled as it does not support openat2 (I did not find much documentation around rosetta2 but it seems to be also using syscall emulation (to some extend at least)).
[0] https://github.com/microsoft/WSL/issues/8203
I've tried different machine image versions (fedora 40/41) and multiple bootc-images but I'm not able to crosscompile (apple silicon to amd64). I'm encountering the same mentioned error messages all the time. Rosetta is turned off, the workaround mentioned in the README doesn't work either.
❯ podman --version
podman version 5.4.0
Traceback (most recent call last):
File "/run/osbuild/bin/org.osbuild.rpm", line 251, in <module>
r = main(args["tree"], args["inputs"], args["options"])
File "/run/osbuild/bin/org.osbuild.rpm", line 207, in main
subprocess.run([
~~~~~~~~~~~~~~^^
"rpm",
^^^^^^
...<8 lines>...
"--install", manifest.name
^^^^^^^^^^^^^^^^^^^^^^^^^^
], cwd=pkgpath, check=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/subprocess.py", line 579, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['rpm', '--verbose', '--root', '/run/osbuild/tree', '--nosignature', '--install', '/tmp/manifest.iajdt8gl']' returned non-zero exit status 254.
Finished module org.osbuild.rpm
Finished pipeline build
manifest - failed
Output:
Failed
@mvo5 @brotaxt
We are still encountering this issue and have posted our output here:
https://github.com/podman-desktop/extension-bootc/issues/1475
Even with Rosetta disabled and the newest podman machine (even tried with fixes...) and it does not compile.
The current solution right now does not work
@mvo5 @brotaxt
Cross posting from https://github.com/osbuild/bootc-image-builder/pull/878 I was able to produce a full log with the debug:
Build Image Log ----------
ID: httpd
Image: registry.gitlab.com/fedora/bootc/examples/httpd
Type: raw
Folder: /Users/cdrage/Downloads/bootc/amd64test
----------
{
"name": "httpd-bootc-image-builder",
"Image": "quay.io/centos-bootc/bootc-image-builder:sha256-c56d3bc64eb551913196e15d9909236b4fefae066f56a50139b4e482dd329ecc",
"Tty": true,
"HostConfig": {
"Privileged": true,
"SecurityOpt": [
"label=type:unconfined_t"
],
"Binds": [
"/Users/cdrage/Downloads/bootc/amd64test:/output/",
"/var/lib/containers/storage:/var/lib/containers/storage"
]
},
"Env": [
"IMAGE_BUILDER_EXPERIMENTAL=debug-qemu-user"
],
"Labels": {
"bootc.image.builder": "true"
},
"Cmd": [
"registry.gitlab.com/fedora/bootc/examples/httpd:latest",
"--output",
"/output/",
"--local",
"--progress",
"verbose",
"--type",
"raw",
"--target-arch",
"amd64"
]
}
----------
podman \
run \
--rm \
--name \
httpd-bootc-image-builder \
--tty \
--privileged \
--security-opt \
label=type:unconfined_t \
-v \
/Users/cdrage/Downloads/bootc/amd64test:/output/ \
-v \
/var/lib/containers/storage:/var/lib/containers/storage \
--label \
bootc.image.builder=true \
quay.io/centos-bootc/bootc-image-builder:sha256-c56d3bc64eb551913196e15d9909236b4fefae066f56a50139b4e482dd329ecc \
registry.gitlab.com/fedora/bootc/examples/httpd:latest \
--output \
/output/ \
--local \
--progress \
verbose \
--type \
raw \
--target-arch \
amd64
----------
WARNING: --local is now the default behavior, you can remove it from the command line
Generating manifest manifest-raw.json
WARNING: target-arch is experimental and needs an installed 'qemu-user' package
Manifest generation step
Done generating manifest
Image building step
Building manifest-raw.json
Build
root: <host>
starting -Pipeline source org.osbuild.containers-storage: 4ce824140f2652c7a630134f41f4cfa7c75b4b9014a57b8fff36c1f8410d6ba0
⏱ Duration: 1744051942s
Pipeline build: 04f145d1d61fc4f65a385e60f60fefadc8f79ce337989282489f0c1113663be3
root: <host>
runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.container-deploy: e35918c01d8f2a00a6369ed6df435a400a717412d651fd9ab95fa0df283aadb7 {
"remove-signatures": true
Build
}
time="2025-04-07T18:52:23Z" level=info msg="Image operating system mismatch: image uses OS \"linux\"+architecture \"amd64\"+\"\", expecting one of \"linux+arm64+\\\"v8\\\", linux+arm64+\\\"\\\"\""
Copying blob sha256:5e913d1742efb874b0ac6178b86615c7748ebf05034210dbd594ac0df92915ba
Copying blob sha256:73f60e7a938b060d07e118ca9b65e80fb2c23849e8926b4cd08c1498d1e7b613
Copying blob sha256:88cd5a620048bfe34abb7d56b98456e8b6a0e766516c34c47b949c90620e2657
Copying blob sha256:9a1a0862c7696bd2e36bf7aad37f9e59a17de5e9ee17e4e7b9e9decc965476e7
Copying blob sha256:4c93572df65cc7405ec56706859b5fc06d61906a3336725662d249b7a09844b3
Copying blob sha256:bc1abff2c8d4d62d4e06e3455d87c243f6620d0439dd0ac1c9d2891758c25f75
Copying blob sha256:0946f19ab19b2488d5188c5f7aabd6b06b2f119384bb2cbae155364b836be74b
Copying blob sha256:40694771f957e0d159533bff1d82515036931bbc226938d8eca7612053d6336d
Copying blob sha256:547b4fc5eb1d0e4ec374f4d01865486ba64df03cf160c6adb2b0d0309c3a50fa
Copying blob sha256:f22f81e8d26786e7986d4b0b44157f7a3f0b7626711e44d6d7531a71b551ac00
Copying blob sha256:76817607bbbad2236f463808a1d62d82b929189fa2091412573dff1813e3a591
Copying blob sha256:ce9898de3cdca7830f8a4508a3071991932493a4fe1182f6463e518bfbf40769
Copying blob sha256:bc4d189d3c07659b399aefca02d7bbe7d1546287a04d0fcab4b94f0dd4c73e0f
Copying blob sha256:1bf396f380e158c4fec306e322f41522548de2181d5390b633abaed5744d5523
Copying blob sha256:fb03d8937a18532ae08cecf2267610f224ac7a287cf100071c9d2698dd7e2b4f
Copying blob sha256:312bd44fbfb86e4a31816751d39104a1bdd0e1ffed486ed5df080bcf92d99885
Copying blob sha256:fb01caf3bc05f9b28511ad89291e78c7c7dbce12eac1015ef47e1c0c528fcadc
Copying blob sha256:7f4854de3164c3df2fa03f8da0039160cda07d6ccf8154fe7a4a1b74d9d2f92b
Copying blob sha256:7cd49ddea5e689996c72a6622ce847e22df580f01ad035fedf35bd8aa7d518c0
Copying blob sha256:d35a6a12ae387e2ac3c1daba618228a450e3f26c91570e85fe39f21f07c2b106
Copying blob sha256:4b55ef76a8a838b4e4d878a9443385486500eb576facf380a55bf299a61f2fef
Copying blob sha256:2a21109c80881e65a3d4dceeea48fd5486053acbbf63be334012fa6291f1b7cd
Copying blob sha256:03f8187d462decd660e9d518a136c79c867bae006e6d4ad0e29cd476b7f81f5d
Copying blob sha256:7fd3632758b032d1f41d535673b314aad8ef42a63eb5eecad56bda93dc344621
Copying blob sha256:7595b08c1e5c903bea93e1e039806aa4bd0fc536d593f80a4f7b428a2d6d2c8a
Copying blob sha256:0cd4ad7d7660acb3eba5a2f052964439610498bbd38df528fafe287256bc0412
Copying blob sha256:ab4bc8650436b4765df9f4a18ea986d076c018238c94f28fee2b0ff698d08055
Copying blob sha256:8bf2dcf18ec5748e2c8e10a6120442946787c797e08371844dfafb69955e396d
Copying blob sha256:92939213b34e6f709b1cda52db187dc773e2805ec72f3de4fab4a5160eb5cfa8
Copying blob sha256:f24cd4a0d082f0fcec67ffa828116885613fbe6b89b624813edcc756a165091f
Copying blob sha256:69b91f1def4779a00343db294a8a0c0a311ee41da671ab64816ef0d87cf7c096
Copying blob sha256:4f9f651da520176b382bce5f60ecf430b593f91e98ea92f9f436f055cba0e1e5
Copying blob sha256:439ee64125955e31f2272baae4d1fe38665301d857748692fc02987eaab86758
Copying blob sha256:a284a4218a2f4decd70ea6ff4bbf202c07aca7e5759a0ab6589cf841dd64ec3c
Copying blob sha256:35663a8f1abeaa5004bcb972a54688f842b38b84682822d5c2e6d87bace92239
Copying blob sha256:42b0545087a2d1fea5aa1ecf140dff4b8189621b52bb078418286cc3949b3fa9
Copying blob sha256:541a0ffd011d6c2fb91f8b8eec2265dd6a2c89cf485a2e84acdc5d9a03142aca
Copying blob sha256:1687f4b9ee26f45cf2cdf513783d81b2c7ef9075be02434d8e7113fe33b35907
Copying blob sha256:eedac9f87b94676e8c440ba212c6ca3e5fa4019370293945361e86072fbec690
Copying blob sha256:f7e76fb6cd935b91d41d10ae89e9e1a6f5f81e42eaa14b3f2d78cc71269306e5
Copying blob sha256:9682812509cfc40d5babc936d144b1d8d7640a94c5b6b48fe89472075d43c7b4
Copying blob sha256:1a9a1ba35aa9d9f9b654f5f77019d273f508187ce5c3bbdd61f79896bf1fc469
Copying blob sha256:7acfbdf59cc4f2939c0cb4379c44c6b5cc19f3337290d534bc5269f9d7f5d662
Copying blob sha256:3c5db1671b4447d58e24130355d288fed4298919ab2a529ffe3e0e0d25e60113
Copying blob sha256:b12a9069a18d531ef4eeecb9ab54fdff3b485531d035dbf766111012a43c6a2a
Copying blob sha256:578a022dfe511d1a03c9b2d886961871e8e5ba9edd08fda25981f71057b5d8a0
Copying blob sha256:def2f208f0ea7742331a581ad00526d524be6020d60d6507bf2caaea658e2270
Copying blob sha256:913c17eb0724357f0908c529db73f4039c08eb4e09632ba94077fc11de1e5795
Copying blob sha256:bf307c12fd75cb17098eacebc434a92c26a3984e55ea2a86f986ecce40a97197
Copying blob sha256:b6964660e8a5f857318d5d85eabb2d5e9b1c13e9a2a5a88e05f2ff85ede78b0b
Copying blob sha256:667633d33e469f29e4eafbea7443f07a13d95d772627ac327e9a369b8df5d457
Copying blob sha256:1e8db2fb37b18c05a0edb844908ae5dd77d37a13b885fc032f459a0826ed7549
Copying blob sha256:32aa02403e107f7ef67537f9319694b5b5850bf2b7f4d2770f4f79302016a948
Copying blob sha256:9452fb8b119fad0f03e4ed9748654998335b086aa4d0a5c51074058c7f7c4f44
Copying blob sha256:e71fdd0e7d669d1f446d90ae1b9335a16c84c9f632c1bc9cbf578b2a3ed138da
Copying blob sha256:97bfae099cc4324a783b143de7663b1f1b10ab305bbcda456cbe51fbed5b7d46
Copying blob sha256:0770d485f40ab2eeb5a6924cdaba2b43f6948d05be217a7c360441e609f02926
Copying blob sha256:3767b7c7b305e576c9ffcfde5608cb18030ca71e7996e48a7e72bdb1ed806a60
Copying blob sha256:c5f8ccb7383009bf077643bf0ea02f0278ae910fd85100722a406a4967923997
Copying blob sha256:dc339a1d578aa42b8c9dca04e1a37a5cd0cd4653e6eae767367d851c537c003c
Copying blob sha256:eb3d4ba8bbd4eb9e9694348bf2106125e01b9cc4a10e62538bbd7cca8d973202
Copying blob sha256:6c824bd82f2198b21921a8a91f32e4762ff8269b65692348acd29772b2f9cefc
Copying blob sha256:f339b9fb8b0544f8cd2933bf2b5d0d5108e91cc0cedc85f4d989f073884ee0e7
Copying blob sha256:cf0b231ef0d07bf8b60198c969cfeac654a0ba18859bedd4f2c88c46915a8987
Copying blob sha256:12787d84fa137cd5649a9005efe98ec9d05ea46245fdc50aecb7dd007f2035b1
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
Copying blob sha256:9b92fda47eddeb39bde21254b7a258bffb19b4f6bd236e80a9cf242a13dc0f16
Copying blob sha256:c1e1d1395db357a458465705c178b609f8423f177a07b4241ca5bb597643b351
Copying config sha256:49d47127dc0d3e07018df945784d1817e9f1d5f9cfbd34743ff0e1f38e466bf2
Writing manifest to image destination
49d47127dc0d3e07018df945784d1817e9f1d5f9cfbd34743ff0e1f38e466bf2
Untagged: docker.io/library/tmp-container-deploy-23283288464956:latest
Deleted: 49d47127dc0d3e07018df945784d1817e9f1d5f9cfbd34743ff0e1f38e466bf2
⏱ Duration: 34s
org.osbuild.selinux: 04f145d1d61fc4f65a385e60f60fefadc8f79ce337989282489f0c1113663be3 {
"file_contexts": "etc/selinux/targeted/contexts/files/file_contexts",
],
"exclude_paths": [
"/usr/bin/mount": "system_u:object_r:install_exec_t:s0",
"labels": {
"/usr/bin/umount": "system_u:object_r:install_exec_t:s0"
"/usr/bin/ostree": "system_u:object_r:install_exec_t:s0",
}
}
setfiles: Regex version mismatch, expected: 10.44 2024-06-07 actual: 10.40 2022-04-14
"/sysroot"
setfiles: Regex version mismatch, expected: 10.44 2024-06-07 actual: 10.40 2022-04-14
setfiles: conflicting specifications for /run/osbuild/tree/usr/lib/systemd/system-generators/bootc-systemd-generator and /run/osbuild/tree/usr/bin/bootc, using system_u:object_r:install_exec_t:s0.
⏱ Duration: 9s
Pipeline image: 7382110d88c0f95f4de88937598a81ee7e9764dba8321b26d89e9c76b98a5df2
Build
root: 04f145d1d61fc4f65a385e60f60fefadc8f79ce337989282489f0c1113663be3
runner: org.osbuild.linux (org.osbuild.linux)
"filename": "disk.raw",
}
"size": "10737418240"
org.osbuild.truncate: 89b68ecb2841a38fda9a6b168b53bd3631822d7c5b2909a20c8e2d051e0800d7 {
⏱ Duration: 1s
org.osbuild.sfdisk: e804b518ffd88b1464351ce758e106faa51897e6c6f59d75dc0e9675e83f6bca {
"uuid": "D209C89E-EA5E-4FBD-B161-B461CCE297E0",
"partitions": [
"label": "gpt",
{
"bootable": true,
"start": 2048,
"type": "21686148-6449-6E6F-744E-656564454649",
"uuid": "FAC7F1FB-3E8D-4137-A512-961DE09A5549"
"size": 1026048,
"start": 4096,
"type": "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",
"uuid": "68B2905B-DF3E-4FB3-80FA-49D1E773AA33"
},
},
{
"size": 2048,
"start": 1030144,
"type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
"uuid": "CB07C243-BC44-4717-853E-28852021225B"
"size": 2097152,
{
{
},
"size": 17844191,
"start": 3127296,
"uuid": "6264D520-3FB9-423F-8AB8-7A0A8E3D3562"
"type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
}
}
device/device (org.osbuild.loopback): loop0 acquired (locked: True)
]
label: gpt
label-id: D209C89E-EA5E-4FBD-B161-B461CCE297E0
start="1030144", size="2097152", type="0FC63DAF-8483-4772-8E79-3D69D8477DE4", uuid="CB07C243-BC44-4717-853E-28852021225B"
start="3127296", size="17844191", type="0FC63DAF-8483-4772-8E79-3D69D8477DE4", uuid="6264D520-3FB9-423F-8AB8-7A0A8E3D3562"
start="4096", size="1026048", type="C12A7328-F81F-11D2-BA4B-00A0C93EC93B", uuid="68B2905B-DF3E-4FB3-80FA-49D1E773AA33"
start="2048", size="2048", type="21686148-6449-6E6F-744E-656564454649", uuid="FAC7F1FB-3E8D-4137-A512-961DE09A5549", bootable
{
"id": "D209C89E-EA5E-4FBD-B161-B461CCE297E0",
"unit": "sectors",
"partitiontable": {
"device": "/dev/loop0",
"label": "gpt",
"firstlba": 2048,
"lastlba": 20971486,
"partitions": [
"sectorsize": 512,
{
"node": "/dev/loop0p1",
"start": 2048,
"uuid": "FAC7F1FB-3E8D-4137-A512-961DE09A5549"
},{
"start": 4096,
"node": "/dev/loop0p2",
"size": 1026048,
"type": "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",
"size": 2048,
"type": "21686148-6449-6E6F-744E-656564454649",
"uuid": "68B2905B-DF3E-4FB3-80FA-49D1E773AA33"
},{
"node": "/dev/loop0p3",
"size": 2097152,
"uuid": "CB07C243-BC44-4717-853E-28852021225B"
},{
"node": "/dev/loop0p4",
"type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
"start": 3127296,
"start": 1030144,
"type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
"size": 17844191,
"uuid": "6264D520-3FB9-423F-8AB8-7A0A8E3D3562"
}
]
}
"volid": "7B7795E7"
}
⏱ Duration: 2s
}
device/device (org.osbuild.loopback): loop0 acquired (locked: True)
org.osbuild.mkfs.fat: 29d9e045519a079d0c763186d976fa56e062572fe24a26b9025cfcc8fc7bb66b {
⏱ Duration: 1s
mkfs.fat 4.2 (2021-01-31)
org.osbuild.mkfs.ext4: 783caf5eaa30bf5eb81f3fe6f62c7569c7899bd4be2a2981a4a233d8c43143d8 {
"uuid": "e7f8604a-460f-4b66-8b09-0db2c5e73931",
"label": "boot"
}
device/device (org.osbuild.loopback): loop0 acquired (locked: True)
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: e7f8604a-460f-4b66-8b09-0db2c5e73931
32768, 98304, 163840, 229376
Superblock backups stored on blocks:
Discarding device blocks: done
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
org.osbuild.mkfs.ext4: 40de1a1f61e6e21c6701f1c27639b943d90c24356bf9efda71caeeff65216299 {
⏱ Duration: 1s
"uuid": "17e68373-4404-49f1-b3f2-31917c96d522",
"label": "root"
device/device (org.osbuild.loopback): loop0 acquired (locked: True)
}
Discarding device blocks: done
Filesystem UUID: 17e68373-4404-49f1-b3f2-31917c96d522
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
mke2fs 1.46.5 (30-Dec-2021)
Superblock backups stored on blocks:
Creating filesystem with 2230523 4k blocks and 558624 inodes
⏱ Duration: 1s
"kernel-args": [
"rw",
org.osbuild.bootc.install-to-filesystem: 91a04092796bdbef1e7ba6834953e1d9fc9f95275d7a47cdd1dc1b34f2a0cbb0 {
"console=ttyS0"
"console=tty0",
],
}
"target-imgref": "registry.gitlab.com/fedora/bootc/examples/httpd:latest"
device/disk (org.osbuild.loopback): loop0 acquired (locked: False)
mount/- (org.osbuild.ext4): mounting /dev/loop0p4 -> /store/tmp/buildroot-tmp-0dh4sa0z/mounts/
mount/boot-efi (org.osbuild.fat): mounting /dev/loop0p2 -> /store/tmp/buildroot-tmp-0dh4sa0z/mounts/boot/efi
mount/boot (org.osbuild.ext4): mounting /dev/loop0p3 -> /store/tmp/buildroot-tmp-0dh4sa0z/mounts/boot
Host kernel does not have SELinux support, but target enables it by default; this is less well tested. See https://github.com/containers/bootc/issues/419
Installing image: docker://registry.gitlab.com/fedora/bootc/examples/httpd:latest
Initializing ostree layout
ostree/deploy/default initialized as OSTree stateroot
Initializing sysroot
Deploying container image...done
ERROR Installing to filesystem: Creating ostree deployment: Performing deployment: Creating importer: skopeo proxy unexpectedly exited during request method Initialize: signal: 6 (SIGABRT) (core dumped)
SIGSEGV: segmentation violation
runtime.netpoll(0xc000040030?)
goroutine 0 [idle]:
/usr/lib/golang/src/runtime/netpoll_epoll.go:165 +0x222 fp=0xffff43ffeb38 sp=0xffff43ffe4a8 pc=0x555555816962
runtime.findRunnable()
runtime.schedule()
/usr/lib/golang/src/runtime/proc.go:3191 +0x825 fp=0xffff43ffec48 sp=0xffff43ffeb38 pc=0x555555822925
PC=0x555555816962 m=5 sigcode=1
/usr/lib/golang/src/runtime/proc.go:3582 +0xb1 fp=0xffff43ffec80 sp=0xffff43ffec48 pc=0x555555823a91
runtime.park_m(0xc000102680?)
/usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0xffff43ffece0 sp=0xffff43ffecc8 pc=0x55555584b690
runtime.mcall()
goroutine 1 [runnable]:
strings.Cut({0x55555664a29b, 0x6}, {0x5555565c5fe0?, 0x1})
/usr/lib/golang/src/strings/strings.go:1278 +0xb4 fp=0xc00073ec20 sp=0xc00073ec18 pc=0x5555558eb214
/usr/lib/golang/src/runtime/proc.go:3745 +0x11f fp=0xffff43ffecc8 sp=0xffff43ffec80 pc=0x555555823f9f
traceback: unexpected SPWRITE function runtime.mcall
encoding/json.parseTag(...)
/usr/lib/golang/src/encoding/json/encode.go:1119 +0xff1 fp=0xc00073f618 sp=0xc00073ec20 pc=0x5555559e2fd1
encoding/json.cachedTypeFields({0x5555568bf968?, 0x5555567899e0})
/usr/lib/golang/src/encoding/json/encode.go:1274 +0x11a fp=0xc00073f6d8 sp=0xc00073f618 pc=0x5555559e3b1a
encoding/json.typeFields({0x5555568bf968, 0x5555567899e0})
/usr/lib/golang/src/encoding/json/tags.go:18
encoding/json.(*decodeState).object(0xc00038b950, {0x5555566850a0?, 0xc000261e00?, 0x100000010?})
encoding/json.(*decodeState).value(0xc00038b950, {0x5555566850a0?, 0xc000261e00?, 0xffffa4bc0108?})
encoding/json.(*decodeState).unmarshal(0xc00038b950, {0x5555566850a0?, 0xc000261e00?})
/usr/lib/golang/src/encoding/json/decode.go:374 +0x3e fp=0xc00073fa28 sp=0xc00073f9b8 pc=0x5555559d649e
/usr/lib/golang/src/encoding/json/decode.go:181 +0x133 fp=0xc00073faa0 sp=0xc00073fa28 pc=0x5555559d5db3
/usr/lib/golang/src/encoding/json/decode.go:108 +0x111 fp=0xc00073fad8 sp=0xc00073faa0 pc=0x5555559d5971
main.(*proxyHandler).processRequest(0xc000124d48?, {0xc0001cc000, 0x21, 0x8000})
encoding/json.Unmarshal({0xc0001cc000, 0x21, 0x8000}, {0x5555566850a0, 0xc000261e00})
main.(*proxyOptions).run(0xc000012888, {0xc00073fc20?, 0x5555559c46a5?, 0x0?}, {0x5555568a37f8?, 0xc000124038?})
/builddir/build/BUILD/containers-skopeo-1c2ab99/cmd/skopeo/proxy.go:868 +0x1b1 fp=0xc00073fbf0 sp=0xc00073fb30 pc=0x555556304211
main.(*proxyOptions).run-fm({0xc000127a30?, 0x0?, 0x0?}, {0x5555568a37f8?, 0xc000124038?})
<autogenerated>:1 +0x45 fp=0xc00073fc30 sp=0xc00073fbf0 pc=0x55555630f3a5
main.proxyCmd.commandAction.func1(0xc0001c1000?, {0xc000127a30, 0x0, 0x1})
github.com/spf13/cobra.(*Command).execute(0xc000161500, {0xc000118290, 0x1, 0x1})
/builddir/build/BUILD/containers-skopeo-1c2ab99/vendor/src/github.com/spf13/cobra/command.go:983 +0xabc fp=0xc00073fe28 sp=0xc00073fc88 pc=0x5555559c759c
/builddir/build/BUILD/containers-skopeo-1c2ab99/cmd/skopeo/utils.go:58 +0x57 fp=0xc00073fc88 sp=0xc00073fc30 pc=0x555556303af7
github.com/spf13/cobra.(*Command).ExecuteC(0xc000004c00)
github.com/spf13/cobra.(*Command).Execute(...)
/builddir/build/BUILD/containers-skopeo-1c2ab99/vendor/src/github.com/spf13/cobra/command.go:1039
/builddir/build/BUILD/containers-skopeo-1c2ab99/vendor/src/github.com/spf13/cobra/command.go:1115 +0x3ff fp=0xc00073ff00 sp=0xc00073fe28 pc=0x5555559c7e5f
/usr/lib/golang/src/encoding/json/decode.go:646 +0x27e fp=0xc00073f9b8 sp=0xc00073f6d8 pc=0x5555559d755e
/builddir/build/BUILD/containers-skopeo-1c2ab99/cmd/skopeo/proxy.go:803 +0x55 fp=0xc00073fb30 sp=0xc00073fad8 pc=0x555556303bf5
runtime.main()
/usr/lib/golang/src/runtime/proc.go:267 +0x2d2 fp=0xc00073ffe0 sp=0xc00073ff40 pc=0x55555581d132
main.main()
runtime.goexit()
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc00005efa8 sp=0xc00005ef88 pc=0x55555581d58e
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00073ffe8 sp=0xc00073ffe0 pc=0x55555584d6a1
runtime.goparkunlock(...)
/builddir/build/BUILD/containers-skopeo-1c2ab99/cmd/skopeo/main.go:131 +0x56 fp=0xc00073ff40 sp=0xc00073ff00 pc=0x5555562fdff6
/usr/lib/golang/src/runtime/proc.go:404
runtime.forcegchelper()
/usr/lib/golang/src/runtime/proc.go:322 +0xb8 fp=0xc00005efe0 sp=0xc00005efa8 pc=0x55555581d418
runtime.goexit()
created by runtime.init.7 in goroutine 1
/usr/lib/golang/src/runtime/proc.go:310 +0x1a
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x55555584d6a1
goroutine 3 [GC sweep wait]:
/usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc00005f778 sp=0xc00005f758 pc=0x55555581d58e
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
/usr/lib/golang/src/runtime/mgcsweep.go:321 +0xdf fp=0xc00005f7c8 sp=0xc00005f778 pc=0x5555558088df
runtime.gcenable.func1()
/usr/lib/golang/src/runtime/mgc.go:200 +0x25 fp=0xc00005f7e0 sp=0xc00005f7c8 pc=0x5555557fd9e5
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005f7e8 sp=0xc00005f7e0 pc=0x55555584d6a1
created by runtime.gcenable in goroutine 1
/usr/lib/golang/src/runtime/mgc.go:200 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime.gopark(0x10000?, 0x5555565c5fb0?, 0x0?, 0x0?, 0x0?)
/usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc00005ff70 sp=0xc00005ff50 pc=0x55555581d58e
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:404
/usr/lib/golang/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00005ffa0 sp=0xc00005ff70 pc=0x555555806129
runtime.(*scavengerState).park(0x5555571121a0)
runtime.bgscavenge(0x0?)
/usr/lib/golang/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00005ffc8 sp=0xc00005ffa0 pc=0x5555558066d9
runtime.gcenable.func2()
/usr/lib/golang/src/runtime/mgc.go:201 +0x25 fp=0xc00005ffe0 sp=0xc00005ffc8 pc=0x5555557fd985
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x55555584d6a1
/usr/lib/golang/src/runtime/mgc.go:201 +0xa5
created by runtime.gcenable in goroutine 1
goroutine 18 [finalizer wait]:
runtime.gopark(0x555556894c80?, 0x15581e701?, 0x0?, 0x0?, 0x5555558257a5?)
/usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc00005e628 sp=0xc00005e608 pc=0x55555581d58e
runtime.runfinq()
/usr/lib/golang/src/runtime/mfinal.go:193 +0x107 fp=0xc00005e7e0 sp=0xc00005e628 pc=0x5555557fc9e7
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005e7e8 sp=0xc00005e7e0 pc=0x55555584d6a1
created by runtime.createfing in goroutine 1
/usr/lib/golang/src/runtime/mfinal.go:163 +0x3d
goroutine 19 [GC worker (idle)]:
/usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc00005a750 sp=0xc00005a730 pc=0x55555581d58e
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime.gcBgMarkWorker()
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x55555584d6a1
/usr/lib/golang/src/runtime/mgc.go:1293 +0xe5 fp=0xc00005a7e0 sp=0xc00005a750 pc=0x5555557ff5a5
/usr/lib/golang/src/runtime/mgc.go:1217 +0x1c
goroutine 34 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime.gcBgMarkWorker()
/usr/lib/golang/src/runtime/mgc.go:1293 +0xe5 fp=0xc00018a7e0 sp=0xc00018a750 pc=0x5555557ff5a5
runtime.goexit()
/usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc00018a750 sp=0xc00018a730 pc=0x55555581d58e
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00018a7e8 sp=0xc00018a7e0 pc=0x55555584d6a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/lib/golang/src/runtime/mgc.go:1217 +0x1c
/usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc00018af50 sp=0xc00018af30 pc=0x55555581d58e
goroutine 35 [GC worker (idle)]:
/usr/lib/golang/src/runtime/mgc.go:1293 +0xe5 fp=0xc00018afe0 sp=0xc00018af50 pc=0x5555557ff5a5
runtime.gcBgMarkWorker()
runtime.gopark(0x328e700993?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00018afe8 sp=0xc00018afe0 pc=0x55555584d6a1
/usr/lib/golang/src/runtime/mgc.go:1217 +0x1c
goroutine 5 [GC worker (idle)]:
/usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000060750 sp=0xc000060730 pc=0x55555581d58e
runtime.gopark(0x328e85c664?, 0x0?, 0x0?, 0x0?, 0x0?)
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/lib/golang/src/runtime/mgc.go:1293 +0xe5 fp=0xc0000607e0 sp=0xc000060750 pc=0x5555557ff5a5
runtime.goexit()
/usr/lib/golang/src/runtime/mgc.go:1217 +0x1c
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime.gopark(0x328ea7f1da?, 0x0?, 0x0?, 0x0?, 0x0?)
goroutine 20 [GC worker (idle)]:
/usr/lib/golang/src/runtime/mgc.go:1293 +0xe5 fp=0xc00005afe0 sp=0xc00005af50 pc=0x5555557ff5a5
/usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc00005af50 sp=0xc00005af30 pc=0x55555581d58e
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005afe8 sp=0xc00005afe0 pc=0x55555584d6a1
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x55555584d6a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/lib/golang/src/runtime/mgc.go:1217 +0x1c
runtime.gopark(0x328eae6976?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime.goexit()
goroutine 6 [GC worker (idle)]:
/usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000060f50 sp=0xc000060f30 pc=0x55555581d58e
runtime.gcBgMarkWorker()
runtime.goexit()
/usr/lib/golang/src/runtime/mgc.go:1293 +0xe5 fp=0xc000060fe0 sp=0xc000060f50 pc=0x5555557ff5a5
/usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000060fe8 sp=0xc000060fe0 pc=0x55555584d6a1
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime.gcBgMarkWorker()
runtime.gcBgMarkWorker()
rax 0x1
rcx 0x0
rbx 0x0
rdx 0xdf8404641
rsi 0xffffffffa4a08e58
rdi 0x555557143ae0
rbp 0xffff43ffeb28
rsp 0xffff43ffe4a8
r9 0x72
r10 0x3
r11 0x0
r8 0xe9
r12 0xffff43ffe528
r13 0x0
r14 0xc0001021a0
r15 0x3
rip 0x555555816962
rflags 0x206
cs 0x33
fs 0x0
gs 0x0
-----
PC=0x55555584f4a1 m=0 sigcode=0
SIGQUIT: quit
goroutine 0 [idle]:
runtime.futex()
/usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0xffffa57ef798 sp=0xffffa57ef790 pc=0x55555584f4a1
runtime.futexsleep(0x555556af3e50?, 0x5707fb20?, 0x555556ddf6e0?)
/usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0xffffa57ef7e8 sp=0xffffa57ef798 pc=0x555555816ad0
runtime.notesleep(0x555557112ea8)
/usr/lib/golang/src/runtime/lock_futex.go:160 +0x87 fp=0xffffa57ef820 sp=0xffffa57ef7e8 pc=0x5555557ef467
/usr/lib/golang/src/runtime/mgc.go:1217 +0x1c
/usr/lib/golang/src/runtime/proc.go:1632
runtime.stopm()
/usr/lib/golang/src/runtime/proc.go:2536 +0x8c fp=0xffffa57ef850 sp=0xffffa57ef820 pc=0x55555582136c
runtime.gcstopm()
/usr/lib/golang/src/runtime/proc.go:2835 +0xaa fp=0xffffa57ef878 sp=0xffffa57ef850 pc=0x555555821f4a
runtime.findRunnable()
/usr/lib/golang/src/runtime/proc.go:2901 +0x3f fp=0xffffa57ef988 sp=0xffffa57ef878 pc=0x55555582213f
runtime.mPark(...)
runtime.schedule()
runtime.goschedImpl(0xc0000061a0)
/usr/lib/golang/src/runtime/proc.go:3582 +0xb1 fp=0xffffa57ef9c0 sp=0xffffa57ef988 pc=0x555555823a91
runtime.gopreempt_m(0x406001?)
/usr/lib/golang/src/runtime/proc.go:3788 +0x4f fp=0xffffa57efa38 sp=0xffffa57ef9f8 pc=0x55555582434f
runtime.newstack()
/usr/lib/golang/src/runtime/stack.go:1074 +0x3cb fp=0xffffa57efbe8 sp=0xffffa57efa38 pc=0x5555558342cb
traceback: unexpected SPWRITE function runtime.morestack
runtime.morestack()
/usr/lib/golang/src/runtime/proc.go:3760 +0xbf fp=0xffffa57ef9f8 sp=0xffffa57ef9c0 pc=0x55555582407f
/usr/lib/golang/src/runtime/asm_amd64.s:593 +0x84 fp=0xffffa57efbf0 sp=0xffffa57efbe8 pc=0x55555584b7e4
rax 0xca
rbx 0x0
rcx 0xffffffffffffffff
rdx 0x0
rdi 0x555557112ea8
rsi 0x80
rbp 0xffffa57ef7d8
rsp 0xffffa57ef790
r8 0x0
r9 0x0
r10 0x0
r11 0x0
r12 0xffffa57ef8f8
r13 0x80
r14 0x555557112740
r15 0x0
rip 0x55555584f4a1
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
-----
SIGQUIT: quit
PC=0x55555584f4a3 m=2 sigcode=0
goroutine 0 [idle]:
runtime.futex()
/usr/lib/golang/src/runtime/sys_linux_amd64.s:558 +0x23 fp=0xffff51efeb90 sp=0xffff51efeb88 pc=0x55555584f4a3
runtime.futexsleep(0x329348c6cb?, 0x0?, 0xffff51efec30?)
/usr/lib/golang/src/runtime/os_linux.go:75 +0xa7 fp=0xffff51efebe0 sp=0xffff51efeb90 pc=0x555555816b47
runtime.notetsleep_internal(0x555557113660, 0xdf8475800)
/usr/lib/golang/src/runtime/lock_futex.go:202 +0xb5 fp=0xffff51efec20 sp=0xffff51efebe0 pc=0x5555557ef595
runtime.notetsleep(0x329347dc6b?, 0xffff00000002?)
/usr/lib/golang/src/runtime/lock_futex.go:225 +0x29 fp=0xffff51efec40 sp=0xffff51efec20 pc=0x5555557ef6c9
runtime.sysmon()
runtime.mstart1()
/usr/lib/golang/src/runtime/proc.go:1600 +0x95 fp=0xffff51efecd0 sp=0xffff51efeca8 pc=0x55555581fd15
runtime.mstart0()
/usr/lib/golang/src/runtime/proc.go:1557 +0x76 fp=0xffff51efed00 sp=0xffff51efecd0 pc=0x55555581fc56
/usr/lib/golang/src/runtime/asm_amd64.s:394 +0x5 fp=0xffff51efed08 sp=0xffff51efed00 pc=0x55555584b605
runtime.mstart()
rbx 0x0
rax 0xfffffffffffffffc
rcx 0xffff51efebc0
rsi 0x80
rbp 0xffff51efebd0
rdi 0x555557113660
rsp 0xffff51efeb88
rdx 0x0
/usr/lib/golang/src/runtime/proc.go:5570 +0x1c9 fp=0xffff51efeca8 sp=0xffff51efec40 pc=0x5555558284a9
r9 0x0
r12 0xffff51efebc0
r10 0xffff51efebc0
r13 0x16
r8 0x0
r11 0x43bbce8438984b5b
r14 0xc0000069c0
rip 0x55555584f4a3
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
-----
SIGQUIT: quit
PC=0x55555584f4a1 m=3 sigcode=0
goroutine 0 [idle]:
runtime.futex()
/usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0xffff516fda80 sp=0xffff516fda78 pc=0x55555584f4a1
/usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0xffff516fdad0 sp=0xffff516fda80 pc=0x555555816ad0
runtime.notesleep(0xc000064948)
runtime.mPark(...)
/usr/lib/golang/src/runtime/proc.go:1632
runtime.stopm()
runtime.futexsleep(0x0?, 0x516fdb00?, 0x555555829707?)
/usr/lib/golang/src/runtime/proc.go:2536 +0x8c fp=0xffff516fdb38 sp=0xffff516fdb08 pc=0x55555582136c
runtime.findRunnable()
/usr/lib/golang/src/runtime/proc.go:3229 +0xb9c fp=0xffff516fdc48 sp=0xffff516fdb38 pc=0x555555822c9c
/usr/lib/golang/src/runtime/lock_futex.go:160 +0x87 fp=0xffff516fdb08 sp=0xffff516fdad0 pc=0x5555557ef467
r15 0x0
runtime.schedule()
runtime.park_m(0xc00051c9c0?)
traceback: unexpected SPWRITE function runtime.mcall
/usr/lib/golang/src/runtime/proc.go:3745 +0x11f fp=0xffff516fdcc8 sp=0xffff516fdc80 pc=0x555555823f9f
/usr/lib/golang/src/runtime/proc.go:3582 +0xb1 fp=0xffff516fdc80 sp=0xffff516fdc48 pc=0x555555823a91
/usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0xffff516fdce0 sp=0xffff516fdcc8 pc=0x55555584b690
rax 0xca
runtime.mcall()
rcx 0xffffffffffffffff
rdx 0x0
rbx 0x0
rbp 0xffff516fdac0
rsp 0xffff516fda78
rsi 0x80
r8 0x0
r9 0x0
r11 0x0
rdi 0xc000064948
r13 0x17f
r10 0x0
r14 0xc000006ea0
r15 0x3
rflags 0x286
rip 0x55555584f4a1
cs 0x33
gs 0x0
r12 0x0
-----
runtime.futex()
SIGQUIT: quit
PC=0x55555584f4a1 m=4 sigcode=0
/usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0xffff50efcb68 sp=0xffff50efcb60 pc=0x55555584f4a1
runtime.futexsleep(0x0?, 0x0?, 0x1?)
fs 0x0
goroutine 0 [idle]:
/usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0xffff50efcbb8 sp=0xffff50efcb68 pc=0x555555816ad0
/usr/lib/golang/src/runtime/lock_futex.go:160 +0x87 fp=0xffff50efcbf0 sp=0xffff50efcbb8 pc=0x5555557ef467
runtime.mPark(...)
/usr/lib/golang/src/runtime/proc.go:1632
runtime.notesleep(0xc000064d48)
runtime.startlockedm(0xc000007520?)
/usr/lib/golang/src/runtime/proc.go:2808 +0x6a fp=0xffff50efcc48 sp=0xffff50efcc20 pc=0x555555821e4a
runtime.stopm()
/usr/lib/golang/src/runtime/proc.go:2536 +0x8c fp=0xffff50efcc20 sp=0xffff50efcbf0 pc=0x55555582136c
runtime.schedule()
/usr/lib/golang/src/runtime/proc.go:3628 +0x6a fp=0xffff50efcc80 sp=0xffff50efcc48 pc=0x555555823a4a
runtime.park_m(0xc000007520?)
runtime.mcall()
traceback: unexpected SPWRITE function runtime.mcall
/usr/lib/golang/src/runtime/proc.go:3745 +0x11f fp=0xffff50efccc8 sp=0xffff50efcc80 pc=0x555555823f9f
rax 0xca
rcx 0xffffffffffffffff
rbx 0x0
rdi 0xc000064d48
/usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0xffff50efcce0 sp=0xffff50efccc8 pc=0x55555584b690
rsi 0x80
rbp 0xffff50efcba8
rsp 0xffff50efcb60
r10 0x0
r13 0x555557077280
r8 0x0
r12 0xffff50efcbb8
rdx 0x0
rip 0x55555584f4a1
rflags 0x286
fs 0x0
cs 0x33
gs 0x0
r9 0x0
r11 0x0
r14 0xc0000071e0
-----
SIGQUIT: quit
PC=0x55555584f4a1 m=6 sigcode=0
r15 0x3
goroutine 0 [idle]:
runtime.futex()
runtime.futexsleep(0xffff437fdc68?, 0x0?, 0x8?)
runtime.notesleep(0x5555571449f8)
/usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0xffff437fdc50 sp=0xffff437fdc00 pc=0x555555816ad0
runtime.templateThread()
/usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0xffff437fdc00 sp=0xffff437fdbf8 pc=0x55555584f4a1
/usr/lib/golang/src/runtime/proc.go:2514 +0x71 fp=0xffff437fdca8 sp=0xffff437fdc88 pc=0x555555821251
runtime.mstart0()
/usr/lib/golang/src/runtime/lock_futex.go:160 +0x87 fp=0xffff437fdc88 sp=0xffff437fdc50 pc=0x5555557ef467
/usr/lib/golang/src/runtime/proc.go:1600 +0x95 fp=0xffff437fdcd0 sp=0xffff437fdca8 pc=0x55555581fd15
/usr/lib/golang/src/runtime/proc.go:1557 +0x76 fp=0xffff437fdd00 sp=0xffff437fdcd0 pc=0x55555581fc56
runtime.mstart()
/usr/lib/golang/src/runtime/asm_amd64.s:394 +0x5 fp=0xffff437fdd08 sp=0xffff437fdd00 pc=0x55555584b605
runtime.mstart1()
rax 0xca
rbx 0x0
rcx 0xffffffffffffffff
rsi 0x80
rsp 0xffff437fdbf8
r9 0x0
rdx 0x0
rdi 0x5555571449f8
r12 0xc0001024e0
r10 0x0
rbp 0xffff437fdc40
r13 0x16
r8 0x0
r15 0x0
cs 0x33
r14 0xc0001024e0
rip 0x55555584f4a1
-----
r11 0x43bbce8438984b5b
fs 0x0
rflags 0x286
gs 0x0
SIGQUIT: quit
PC=0x55555584f4a1 m=7 sigcode=0
goroutine 0 [idle]:
runtime.futex()
/usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0xffff429fea80 sp=0xffff429fea78 pc=0x55555584f4a1
runtime.notesleep(0xc000190148)
/usr/lib/golang/src/runtime/lock_futex.go:160 +0x87 fp=0xffff429feb08 sp=0xffff429fead0 pc=0x5555557ef467
/usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0xffff429fead0 sp=0xffff429fea80 pc=0x555555816ad0
runtime.futexsleep(0xd9?, 0x429feb00?, 0x555555829707?)
/usr/lib/golang/src/runtime/proc.go:1632
runtime.mPark(...)
runtime.schedule()
runtime.stopm()
runtime.findRunnable()
/usr/lib/golang/src/runtime/proc.go:3582 +0xb1 fp=0xffff429fec80 sp=0xffff429fec48 pc=0x555555823a91
/usr/lib/golang/src/runtime/proc.go:2536 +0x8c fp=0xffff429feb38 sp=0xffff429feb08 pc=0x55555582136c
/usr/lib/golang/src/runtime/proc.go:3229 +0xb9c fp=0xffff429fec48 sp=0xffff429feb38 pc=0x555555822c9c
/usr/lib/golang/src/runtime/proc.go:3745 +0x11f fp=0xffff429fecc8 sp=0xffff429fec80 pc=0x555555823f9f
runtime.park_m(0xc000007860?)
traceback: unexpected SPWRITE function runtime.mcall
/usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0xffff429fece0 sp=0xffff429fecc8 pc=0x55555584b690
rax 0xca
rdx 0x0
rbx 0x0
rcx 0xffffffffffffffff
rdi 0xc000190148
rbp 0xffff429feac0
rsp 0xffff429fea78
r8 0x0
rsi 0x80
runtime.mcall()
r10 0x0
r9 0x0
r12 0x0
r13 0x1
r14 0xc0001844e0
r15 0x1
r11 0x0
cs 0x33
rip 0x55555584f4a1
fs 0x0
rflags 0x286
gs 0x0
-----
SIGQUIT: quit
PC=0x55555584f4a1 m=8 sigcode=0
goroutine 0 [idle]:
runtime.futexsleep(0x555555800765?, 0x421fdb00?, 0x555555829707?)
/usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0xffff421fda80 sp=0xffff421fda78 pc=0x55555584f4a1
/usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0xffff421fdad0 sp=0xffff421fda80 pc=0x555555816ad0
runtime.futex()
runtime.notesleep(0xc000600148)
runtime.mPark(...)
/usr/lib/golang/src/runtime/lock_futex.go:160 +0x87 fp=0xffff421fdb08 sp=0xffff421fdad0 pc=0x5555557ef467
/usr/lib/golang/src/runtime/proc.go:2536 +0x8c fp=0xffff421fdb38 sp=0xffff421fdb08 pc=0x55555582136c
runtime.findRunnable()
/usr/lib/golang/src/runtime/proc.go:3229 +0xb9c fp=0xffff421fdc48 sp=0xffff421fdb38 pc=0x555555822c9c
runtime.schedule()
runtime.stopm()
/usr/lib/golang/src/runtime/proc.go:1632
runtime.park_m(0xc0001841a0?)
/usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0xffff421fdce0 sp=0xffff421fdcc8 pc=0x55555584b690
/usr/lib/golang/src/runtime/proc.go:3582 +0xb1 fp=0xffff421fdc80 sp=0xffff421fdc48 pc=0x555555823a91
runtime.mcall()
/usr/lib/golang/src/runtime/proc.go:3745 +0x11f fp=0xffff421fdcc8 sp=0xffff421fdc80 pc=0x555555823f9f
traceback: unexpected SPWRITE function runtime.mcall
rax 0xca
rcx 0xffffffffffffffff
rdi 0xc000600148
rdx 0x0
r8 0x0
rsp 0xffff421fda78
rbp 0xffff421fdac0
r11 0x0
r9 0x0
r10 0x0
r13 0x1
r12 0x0
r14 0xc0006021a0
rsi 0x80
rip 0x55555584f4a1
rbx 0x0
rflags 0x286
cs 0x33
r15 0x1
fs 0x0
gs 0x0
-----
PC=0x55555584f4a1 m=9 sigcode=0
goroutine 0 [idle]:
runtime.futex()
SIGQUIT: quit
/usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0xffff419fca80 sp=0xffff419fca78 pc=0x55555584f4a1
runtime.futexsleep(0xd9?, 0x419fcb00?, 0x555555829707?)
/usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0xffff419fcad0 sp=0xffff419fca80 pc=0x555555816ad0
/usr/lib/golang/src/runtime/lock_futex.go:160 +0x87 fp=0xffff419fcb08 sp=0xffff419fcad0 pc=0x5555557ef467
runtime.notesleep(0xc000680148)
/usr/lib/golang/src/runtime/proc.go:2536 +0x8c fp=0xffff419fcb38 sp=0xffff419fcb08 pc=0x55555582136c
runtime.stopm()
/usr/lib/golang/src/runtime/proc.go:3229 +0xb9c fp=0xffff419fcc48 sp=0xffff419fcb38 pc=0x555555822c9c
runtime.schedule()
/usr/lib/golang/src/runtime/proc.go:1632
runtime.mPark(...)
runtime.findRunnable()
/usr/lib/golang/src/runtime/proc.go:3582 +0xb1 fp=0xffff419fcc80 sp=0xffff419fcc48 pc=0x555555823a91
runtime.park_m(0xc000007860?)
runtime.mcall()
traceback: unexpected SPWRITE function runtime.mcall
rbx 0x0
/usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0xffff419fcce0 sp=0xffff419fccc8 pc=0x55555584b690
rsi 0x80
/usr/lib/golang/src/runtime/proc.go:3745 +0x11f fp=0xffff419fccc8 sp=0xffff419fcc80 pc=0x555555823f9f
rdx 0x0
rdi 0xc000680148
rbp 0xffff419fcac0
rsp 0xffff419fca78
rcx 0xffffffffffffffff
rax 0xca
r9 0x0
r11 0x0
r12 0x1
r10 0x0
r14 0xc0006821a0
r13 0x2
r8 0x0
r15 0x1
cs 0x33
rip 0x55555584f4a1
fs 0x0
gs 0x0
qemu: uncaught target signal 6 (Aborted) - core dumped
rflags 0x286
File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in <module>
File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main
subprocess.run(pargs, env=env, check=True)
File "/usr/lib64/python3.9/subprocess.py", line 528, in run
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]49d47127dc0d3e07018df945784d1817e9f1d5f9cfbd34743ff0e1f38e466bf2', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--target-imgref', 'registry.gitlab.com/fedora/bootc/examples/httpd:latest', '/run/osbuild/mounts']' returned non-zero exit status 1.
Traceback (most recent call last):
raise CalledProcessError(retcode, process.args,
r = main(args["options"], args["inputs"], args["paths"])
mount/boot (org.osbuild.ext4): umount: /store/tmp/buildroot-tmp-0dh4sa0z/mounts/boot unmounted
mount/boot-efi (org.osbuild.fat): umount: /store/tmp/buildroot-tmp-0dh4sa0z/mounts/boot/efi unmounted
manifest - failed
mount/- (org.osbuild.ext4): umount: /store/tmp/buildroot-tmp-0dh4sa0z/mounts/ unmounted
Failed
⏱ Duration: 5s
2025/04/07 18:53:23 error: cannot run osbuild: error running osbuild: exit status 1
Container exited with a non-zero exit code.
It looks like this might be https://gitlab.com/qemu-project/qemu/-/issues/2027 - the backtrace is slightly different but the reproducer is essentially just x509.SystemCertPool() which is also called in skopeo (indirectly via `vendor/github.com/containers/image/v5/pkg/tlsclientconfig/tlsclientconfig.go). The issue there is that the host page size is different from the guest (qemu-static-x86_64) page size, what is the output of:
$ podman machine start
$ podman machine ssh getconf PAGESIZE
?
It looks like this might be https://gitlab.com/qemu-project/qemu/-/issues/2027 - the backtrace is slightly different but the reproducer is essentially just
x509.SystemCertPool()which is also called in skopeo (indirectly via `vendor/github.com/containers/image/v5/pkg/tlsclientconfig/tlsclientconfig.go). The issue there is that the host page size is different from the guest (qemu-static-x86_64) page size, what is the output of:$ podman machine start $ podman machine ssh getconf PAGESIZE ?
It's 4096 for the page size:
▶ podman machine start
Starting machine "podman-machine-default"
API forwarding listening on: /var/folders/cn/pm5mgrcs5h339nphypxtvd080000gn/T/podman/podman-machine-default-api.sock
The system helper service is not installed; the default Docker API socket
address can't be used by podman. If you would like to install it, run the following commands:
sudo /opt/podman/bin/podman-mac-helper install
podman machine stop; podman machine start
You can still connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:
export DOCKER_HOST='unix:///var/folders/cn/pm5mgrcs5h339nphypxtvd080000gn/T/podman/podman-machine-default-api.sock'
Machine "podman-machine-default" started successfully
~
▶ podman machine ssh getconf PAGESIZE
4096
Thanks! Then its not the different page-sizes, that is good to know. Reading the backtrace closely it looks like it dies in skopeo experimental-image-proxy so it might be nice to try to reproduce in isolation to see if just running this is enough (it needs some setup though first it seems, just doing it naively on my machine gives me:
$ skopeo experimental-image-proxy
FATA[0000] file file+net sock: getsockopt: socket operation on non-socket
```
The problem is the intersection of qemu userspace emulation with bootc install (and nontrivial code run in userspace). I think where we want to go in the medium term is to stop having bootc-image-builder set up userspace emulation for this case.
Instead for cross-arch I think we should change the tooling to launch full-system qemu emulation. One way to do that could be to have podman support launching (via full emulation) another instance of podman-machine. But what starts to get really messy is we need these two virtual machines to share container storage. Which is maybe where https://github.com/cgwalters/cstor-dist comes into play...or we try to push hard to store container images on the host (like the original krun-based podman machine is).
That said, for use cases that have a trivial partition layout, going with a systemd-repart style approach would let us bypass the first part (emulated podman machine). We still need it for the second part though anyways.