ostree container image pull to bare-user repo silent metadata corruption & non reproducibility

[champtar]

This is with rpm-ostree-2025.5-1.el9.x86_64, will need to redo the tests with main at some point

Trying to use ostree container image pull in a container (for my CI) with all modes of ostree repo:

• bare (in container): Importing regfile small: Writing content object: Setting xattrs: fsetxattr(security.selinux): Invalid argument
• bare-user: seems to work but silent corruption ...
• bare-split-xattrs: Importing regfile small: Writing content object: Not allowed due to repo mode
• archive: Importing regfile: Cannot currently use ostree_repo_write_regfile() on an archive mode repository

# compose an image
# rpm-ostree compose image --initialize-mode=always --cachedir=build/aaa -l org.opencontainers.image.version=my-version build/ostree-tmp-ba/ba.yaml build/test.oci
...
Wrote commit: 67235f74d871ad8803dabdb58bdb13b222bcd20d679f7ebe0c2c19cf0a82d178
Pushed digest: sha256:0dca95e35c6d4925d537e8713b8e1c5f13f747f3ad8c9b57cec05a21da49ebca

# rm -rf build/bbb                                                                                                                                                  
# ostree init --mode=bare-user --repo=build/bbb                                                                                                                     
# ostree container image pull build/bbb ostree-unverified-image:oci-archive:build/test.oci                                                                          
layers already present: 0; layers needed: 65 (5.5 GB)
 2.13 MiB [████████████████████] (0s) Fetched ostree chunk 91f16f4fb72a6a5d137                                                                                                              
Wrote: ostree-unverified-image:oci-archive:build/test.oci => cf9fb0c3e462e7c8cb0c6c4ed93fd6d9af32bd8b200947c8b0a5cfdd19417b2c
# ostree diff --repo=build/bbb 67235f74d871ad8803dabdb58bdb13b222bcd20d679f7ebe0c2c19cf0a82d178 cf9fb0c3e462e7c8cb0c6c4ed93fd6d9af32bd8b200947c8b0a5cfdd19417b2c
M    /usr/bin/sudo
M    /usr/etc/gshadow
M    /usr/etc/shadow
M    /usr/etc/polkit-1/rules.d
M    /usr/libexec/utempter
M    /usr/share/polkit-1/rules.d

# ostree ls --repo=build/bbb -C -X 67235f74d871ad8803dabdb58bdb13b222bcd20d679f7ebe0c2c19cf0a82d178 /usr/bin/sudo /usr/etc/gshadow /usr/etc/shadow /usr/etc/polkit-1/rules.d /usr/libexec/utempter /usr/share/polkit-1/rules.d
-04111 0 0 185304 94bac51f61dbc11af379382273260526c5aee95020f32810c5967b553430c034 { [(b'security.selinux', b'system_u:object_r:sudo_exec_t:s0')] } /usr/bin/sudo
-00000 0 0    459 ce76e70a2af5bf09fd10e825dee9b294c3d35921e0a815539b69b3d94d0974af { [(b'security.selinux', b'system_u:object_r:shadow_t:s0')] } /usr/etc/gshadow
-00000 0 0    604 3a7e2261fe7ad31ca7b7993b24fd6eee19fb2db60d05b206e4bf83be49f2b28d { [(b'security.selinux', b'system_u:object_r:shadow_t:s0')] } /usr/etc/shadow
d00700 401 0      0 6d5bd00e1698c3824b0497ca1e41847635e954b80ef11f430e2926afaeca5371 efd83da08a1ed1c23874884143fdd917304faaf7ffb018aeeb34cbb7d854cdab { [(b'security.selinux', b'system_u:object_r:etc_t:s0')] } /usr/etc/polkit-1/rules.d
-00644 0 0    326 e2288053a5a910788f2813ea426e1d2ef733546b8f6a64d2167b9d90166052af { [(b'security.selinux', b'system_u:object_r:etc_t:s0')] } /usr/etc/polkit-1/rules.d/50-default.rules
d00755 0 35      0 79d4eac95c0f499dd439a8ce3fbff482ed5961df7a415d1c76167c714aa5fad4 2bbed40a44ba182c6a790ddac9db23f3b8808f95be99057338a52317a12b0065 { [(b'security.selinux', b'system_u:object_r:bin_t:s0')] } /usr/libexec/utempter
-02711 0 22  16072 e9fb00b48a01cd32e4b164ec3ce830f185e9a129f41ac3a7abb02f9be6fd107f { [(b'security.selinux', b'system_u:object_r:utempter_exec_t:s0')] } /usr/libexec/utempter/utempter
d00700 401 0      0 0c78500188f961b19ced648c5d9e8448deeba06ff75c09e00d996173c26e82d3 053d66cee0b436fec525542392cb5495dfa632fbd8f24e9ee0f204743b9ccb81 { [(b'security.selinux', b'system_u:object_r:usr_t:s0')] } /usr/share/polkit-1/rules.d
-00644 0 0    252 d405005b55d781313ebc0d9e994303b14831e446e8bcac837c7a20d213f84555 { [(b'security.selinux', b'system_u:object_r:usr_t:s0')] } /usr/share/polkit-1/rules.d/org.freedesktop.fwupd.rules

# ostree ls --repo=build/bbb -C -X cf9fb0c3e462e7c8cb0c6c4ed93fd6d9af32bd8b200947c8b0a5cfdd19417b2c /usr/bin/sudo /usr/etc/gshadow /usr/etc/shadow /usr/etc/polkit-1/rules.d /usr/libexec/utempter /usr/share/polkit-1/rules.d
-00511 0 0 185304 7205ee7ed7903fe25e0d1e342fbca219f9047e2904124f28df2516c76d68d0eb { [(b'security.selinux', b'system_u:object_r:sudo_exec_t:s0')] } /usr/bin/sudo
-00400 0 0    459 a74defb9b361ba642d5d0e16b1213477e9d0999d5a8e616ef3922ab7a2984c75 { [(b'security.selinux', b'system_u:object_r:shadow_t:s0')] } /usr/etc/gshadow
-00400 0 0    604 42481c4f6d2aeb731551e3498deee4abe8cd1fb8a4d4445b111c87f394b4dbfa { [(b'security.selinux', b'system_u:object_r:shadow_t:s0')] } /usr/etc/shadow
d00700 0 0      0 6d5bd00e1698c3824b0497ca1e41847635e954b80ef11f430e2926afaeca5371 b111bcbcb435b1419e0c4ce40a637f4a88bed2989cdad4503f3b9c09ba6c39b5 { [(b'security.selinux', b'system_u:object_r:etc_t:s0')] } /usr/etc/polkit-1/rules.d
-00644 0 0    326 e2288053a5a910788f2813ea426e1d2ef733546b8f6a64d2167b9d90166052af { [(b'security.selinux', b'system_u:object_r:etc_t:s0')] } /usr/etc/polkit-1/rules.d/50-default.rules
d00755 0 0      0 79d4eac95c0f499dd439a8ce3fbff482ed5961df7a415d1c76167c714aa5fad4 249ffc1afa92a53d0abff59d378dda0ff0a44bc7425299a7b398e832812276a3 { [(b'security.selinux', b'system_u:object_r:bin_t:s0')] } /usr/libexec/utempter
-02711 0 22  16072 e9fb00b48a01cd32e4b164ec3ce830f185e9a129f41ac3a7abb02f9be6fd107f { [(b'security.selinux', b'system_u:object_r:utempter_exec_t:s0')] } /usr/libexec/utempter/utempter
d00700 0 0      0 0c78500188f961b19ced648c5d9e8448deeba06ff75c09e00d996173c26e82d3 721a1ad087975fbbf676ee671ede0f0835b7b3849e760834513c051bd64c0df9 { [(b'security.selinux', b'system_u:object_r:usr_t:s0')] } /usr/share/polkit-1/rules.d
-00644 0 0    252 d405005b55d781313ebc0d9e994303b14831e446e8bcac837c7a20d213f84555 { [(b'security.selinux', b'system_u:object_r:usr_t:s0')] } /usr/share/polkit-1/rules.d/org.freedesktop.fwupd.rules

000 is defaulted to 400, setuid is dropped, if user/group is missing in the local os it defaults to 0 (current user)

I know unencapsulate exists, I'm testing pull without layer first

As a bonus, ostree container image pull commit are not reproducible

# ostree log --repo=build/bbb cf9fb0c3e462e7c8cb0c6c4ed93fd6d9af32bd8b200947c8b0a5cfdd19417b2c                                                                      
commit cf9fb0c3e462e7c8cb0c6c4ed93fd6d9af32bd8b200947c8b0a5cfdd19417b2c
ContentChecksum:  725525333aa4d26fe56560c71ed47c846228bc90c61e7424c500c837b119e5e0
Date:  2025-05-30 20:41:44 +0000
(no subject)

# rm -rf build/bbb                                                                                                                                                  
# ostree init --mode=bare-user --repo=build/bbb                                                                                                                     
# ostree container image pull build/bbb ostree-unverified-image:oci-archive:build/test.oci                                                                          
layers already present: 0; layers needed: 65 (5.5 GB)
 2.13 MiB [████████████████████] (0s) Fetched ostree chunk 91f16f4fb72a6a5d137                                                                                                              
Wrote: ostree-unverified-image:oci-archive:build/test.oci => 5d66b36e6d2709a9ad94ac87a68767aa5fc6a276ca73745179118d1a25b173e7
 
# ostree log --repo=build/bbb 5d66b36e6d2709a9ad94ac87a68767aa5fc6a276ca73745179118d1a25b173e7
commit 5d66b36e6d2709a9ad94ac87a68767aa5fc6a276ca73745179118d1a25b173e7
ContentChecksum:  725525333aa4d26fe56560c71ed47c846228bc90c61e7424c500c837b119e5e0
Date:  2025-05-30 20:41:44 +0000
(no subject)

But ContentChecksum is the same so not too bad

[champtar]

With --mode=bare as root outside container (else it fails with SELinux) / ContentChecksum is identical, but commit still change

[champtar]

For context I initially wanted to do:

ostree container image pull 
rpm-ostree compose container-encapsulate

This was before I found rpm-ostree compose build-chunked-oci

So now I'm just wondering if we should be able to pull to bare-user repo, or if we should error out for anything other than bare repo

[cgwalters]

This is with rpm-ostree-2025.5-1.el9.x86_64, will need to redo the tests with main at some point

Just FYI we are planning to do https://issues.redhat.com/browse/RHEL-72863 in 9.7 which will mean the implementation code moves to bootc (it's already that way in rhel10, and only didn't happen in 9.6 by mistake)

[cgwalters]

bare (in container): Importing regfile small: Writing content object: Setting xattrs: fsetxattr(security.selinux): Invalid argument

It's because of install_t...soooo annoying. See the giant dance in https://github.com/bootc-dev/bootc/blob/5abda6a5914ecbe534f9c2eedf9483078f37d814/lib/src/lsm.rs#L57 - the problem here I think is that code may not be being invoked in this path.

bare-user: seems to work but silent corruption ...

Hmm that is concerning indeed, offhand looks like file caps security.capability xattr and non-root ownership

[jlebon]

As a bonus, ostree container image pull commit are not reproducible

OK, I think I addressed that part at least in https://github.com/bootc-dev/bootc/pull/1421.

[champtar]

There was also #1347 and #1339

[ericcurtin]

Hit something similar:

$ sudo ./target/debug/bootc switch --transport oci-archive /var/tmp/bootc-image.tar (Try the bootc switch with oci-archive transport again) 
layers already present: 63; layers needed: 14 (24.9 GB)
error: Switching: Switching (ostree): Pulling: Importing: Unencapsulating base: Layer
sha256:e2eee3f1f87e86c2ed0d56e937e94c64e95d29a31272df97b3414cfa87ea1f47: Importing objects: Importing object
17/091a2697d5ef33418eb454ff86fe2046ec6ec70edd49f987eaa0b82490c07f.file: Processing content object
17091a2697d5ef33418eb454ff86fe2046ec6ec70edd49f987eaa0b82490c07f: Importing regfile: Setting xattrs: fsetxattr(security.selinux): Invalid argument

[ericcurtin]

It's the same with --transport oci of course

[champtar]

@ericcurtin you likely just have the wrong context on your debug binary, it needs to be install_t

[ericcurtin]

@champtar thanks for the prompt response, you are a lifesaver, giving it a whirl

[ericcurtin]

This might do it for development I guess:

sudo bootc usroverlay sudo cp ./target/debug/bootc /usr/sbin/bootc

[champtar]

@ericcurtin you can also use chcon --reference

[ericcurtin]

Thanks, I got it done anyway:

https://github.com/bootc-dev/bootc/pull/1776

Someday I'll get better at working with selinux :)