podman icon indicating copy to clipboard operation
podman copied to clipboard

[v4.4.1-rhel] Bump runc to 1.2.9, Buildah to v1.29.6

Open TomSweeneyRedHat opened this issue 3 months ago • 13 comments

Addresses CVE-2025-52881, and also CVE-2025-31133 and CVE-2025-52565

Fixes: https://issues.redhat.com/browse/RHEL-126910, https://issues.redhat.com/browse/RHEL-126908, https://issues.redhat.com/browse/OCPBUGS-64908, https://issues.redhat.com/browse/OCPBUGS-64909, https://issues.redhat.com/browse/OCPBUGS-64907

Checklist

Ensure you have completed the following checklist for your pull request to be reviewed:

  • [x] Certify you wrote the patch or otherwise have the right to pass it on as an open-source patch by signing all commits. (git commit -s). (If needed, use git commit -s --amend). The author email must match the sign-off email address. See CONTRIBUTING.md for more information.
  • [x] Referenced issues using Fixes: #00000 in commit message (if applicable)
  • [x] Tests have been added/updated (or no tests are needed)
  • [x] Documentation has been updated (or no documentation changes are needed)
  • [x] All commits pass make validatepr (format/lint checks)
  • [x] Release note entered in the section below (or None if no user-facing changes)

Does this PR introduce a user-facing change?

None

TomSweeneyRedHat avatar Dec 12 '25 23:12 TomSweeneyRedHat

@timcoding1988 I'm assuming I need to bump AWS credentials here too? If not, please holler.

�[1;33mActivated AWS CLI for service acount.�[0m

An error occurred (AuthFailure) when calling the DescribeImages operation: AWS was not able to validate the provided access credentials
�[1;31mERROR: Failed to update one or more image timestamps:  fedora-podman-aws-arm64-c6447802205601792�[0m

TomSweeneyRedHat avatar Dec 17 '25 15:12 TomSweeneyRedHat

@timcoding1988 I'm assuming I need to bump AWS credentials here too? If not, please holler.


�[1;33mActivated AWS CLI for service acount.�[0m



An error occurred (AuthFailure) when calling the DescribeImages operation: AWS was not able to validate the provided access credentials

�[1;31mERROR: Failed to update one or more image timestamps:  fedora-podman-aws-arm64-c6447802205601792�[0m

@TomSweeneyRedHat correct

timcoding1988 avatar Dec 17 '25 19:12 timcoding1988

@lsm5 @dashea is this ready to go forward?

TomSweeneyRedHat avatar Jan 05 '26 23:01 TomSweeneyRedHat

@lsm5 do you know the status of this PR versus the branch you've been working on?

dashea avatar Jan 13 '26 20:01 dashea

@lsm5 do you know the status of this PR versus the branch you've been working on?

I'll update this PR with my updates.

lsm5 avatar Jan 14 '26 13:01 lsm5

Just to double-check, let me see if I can get this through the RHEL build and system tests internally...

cevich avatar Jan 15 '26 16:01 cevich

...getting an rpmbuild compile error for podman, attempting to debug...

cevich avatar Jan 15 '26 17:01 cevich

debug fail :disappointed: Maybe it's not worth holding up this PR?

In either case, what I'm seeing is what appears like a normal rpmbuild but it simply dies at the end:

...cut...
github.com/containers/podman/vendor/github.com/containers/common/libimage
github.com/containers/podman/vendor/github.com/containers/buildah/internal/util
github.com/containers/podman/vendor/github.com/containers/buildah/util
github.com/containers/podman/vendor/github.com/containers/buildah/bind
github.com/containers/podman/vendor/github.com/containers/buildah/copier
github.com/containers/podman/vendor/github.com/containers/buildah/internal/parse
github.com/containers/podman/vendor/github.com/containers/buildah/chroot
github.com/containers/podman/vendor/github.com/containers/buildah/pkg/parse
github.com/containers/podman/vendor/github.com/containers/buildah/pkg/cli
github.com/containers/podman/vendor/github.com/containers/buildah
github.com/containers/podman/vendor/github.com/containers/buildah/imagebuildah
error: Bad exit status from /var/tmp/rpm-tmp.wyeZlc (%build)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.wyeZlc (%build)

Before the build, I updated all the relevant golang packages up to 1.25.3. There are no other errors displayed, just a handful of code warnings here and there, nothing that looks serious. So I'm unsure of where to look or what to try to fix it :face_with_diagonal_mouth:

cevich avatar Jan 15 '26 18:01 cevich

Update: Instead of doing an rpmbuild on my RHEL 9.2 VM (w/ updated golang), I just tried building directly from the 2e8bce2 source. This is what I got:

[root@cevich-1mt-rhel-9 containers-podman-2e8bce2]# make
fatal: not a git repository (or any of the parent directories): .git
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build \
         \
        -ldflags ' -X github.com/containers/podman/v4/libpod/define.buildInfo=1768507185 -X github.com/containers/podman/v4/libpod/config._installPrefix=/usr/local -X github.com/containers/podman/v4/libpod/config._etcDir=/usr/local/etc -X github.com/containers/podman/v4/pkg/systemd/quadlet._binDir=/usr/local/bin -X github.com/containers/common/pkg/config.additionalHelperBinariesDir= ' \
        -tags " exclude_graphdriver_btrfs btrfs_noversion selinux systemd libsubid exclude_graphdriver_devicemapper seccomp" \
        -o bin/podman ./cmd/podman
# github.com/containers/podman/v4/libpod/define
libpod/define/errors.go:96:19: undefined: util.ErrDetach
# github.com/containers/podman/v4/utils
utils/utils_supported.go:29:23: undefined: cgroups.GetUserConnection
make: *** [Makefile:321: bin/podman] Error 1
[root@cevich-1mt-rhel-9 containers-podman-2e8bce2]# go version
go version go1.25.3 (Red Hat 1.25.3-2.el9) linux/amd64

cevich avatar Jan 15 '26 20:01 cevich

Indeed, a # grep 'ErrDetach' vendor/github.com/containers/common/pkg/util/* gives nothing. Hrmmmm. I tried running make vendor-in-container but it didn't change anything. I'm at a loss why it compiles fine on F41 in CI, but fails on RHEL 9.2 :confused:

cevich avatar Jan 15 '26 20:01 cevich

Ugh, -sigh-, no, silly me, this is all my fault. I took the top commit ID from the list in github instead of the bottom one (e581a2919). Sorry for the false alarm and noise.

It's building fine on RHEL 9.2 and I've got the BATS tests running now.

I'm going into the corner to cry while I wait.

Edit: Update, BATS Tests appear to be running fine, 99% passing w/ only one expected failure so far.

cevich avatar Jan 15 '26 20:01 cevich

Final manual results:

  • rootless failures:
    • many sigproxy fails (frequent flaker)
    • podman ps --external (exit 0 expected 2)
    • pasta (not present; expected)
  • remote rootless:
    • many sigproxy (frequent flaker)
    • podman ps --external (exit 0, expected 2) remote root:
    • podman-remote: non-default connection # podman-remote --remote ps (negative test; exit 0, exp. 125)

The multiple podman ps failures are a bit troubling, but given the other wise high PASS-rate I'm inclined to ignore them. So my overall opinion is this LGTM. Whew!

cevich avatar Jan 16 '26 14:01 cevich

@lsm5 I was working on backporting commits from here to v4.2.0-rhel and noticed in the "Fix multiple podman cp issues for podman-remote" commit, there are a bunch of documentation changes not mentioned in the commit. I'm seeing addition of pages for podman artifacts and podman farm related, neither of which are supported in v4.4

Would you mind double-checking if these are intended to be in that commit/this PR?

cevich avatar Jan 16 '26 19:01 cevich

@lsm5 I was working on backporting commits from here to v4.2.0-rhel and noticed in the "Fix multiple podman cp issues for podman-remote" commit, there are a bunch of documentation changes not mentioned in the commit. I'm seeing addition of pages for podman artifacts and podman farm related, neither of which are supported in v4.4

Would you mind double-checking if these are intended to be in that commit/this PR?

cleaned up. PTAL.

lsm5 avatar Jan 19 '26 11:01 lsm5

Thanks Lokesh, that looks better now.

cevich avatar Jan 19 '26 14:01 cevich

fwiw, podman ps --external passed consistently for me on RHEL-9.2.0-Nightly compose on testing-farm internal ranch.

lsm5 avatar Jan 19 '26 15:01 lsm5

@cevich @TomSweeneyRedHat PTAL on taking this further / merging.

lsm5 avatar Jan 20 '26 12:01 lsm5

The changes compile on RHEL 9.2 and at least the local-root system tests pass (except for one very minor SELinux test-code bug I'm ignoring).

The local-rootless system test failed on the SIGPROXY tests (frequent flakers), many pasta tests (environment doesn't support IPv6), and again :thinking: the "podman ps --external".

The ps external failure is happening right near the end of the test. It's expecting a "buildah created" container (created by special buildah command) to not be removable w/o a "-f", but it gets removed anyway:

not ok 115 podman ps --external
# (from function `die' in file ./helpers.bash, line 558,
#  from function `run_podman' in file ./helpers.bash, line 228,
#  in test file ./040-ps.bats, line 147)
#   `run_podman 2 rm "$cid"' failed
# $ podman rm -t 0 --all --force --ignore
...cut...
# $ podman ps --external
# CONTAINER ID  IMAGE                              COMMAND     CREATED
  STATUS      PORTS       NAMES
# 140a22894a2e  quay.io/libpod/testimage:20221018  buildah     Less than a second ago
  Storage                 testimage-working-container
# $ podman rm 140a22894a2e
# 140a22894a2e
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #| FAIL: exit code is 0; expected 2
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# # [teardown]
# $ podman pod rm -t 0 --all --force --ignore
# $ podman rm -t 0 --all --force --ignore
# $ podman network prune --force
# $ podman volume rm -a -f

I'm not entirely familiar with this test, but based on the results when run on RHEL 9.2, it seems like either:

  1. The earlier run_podman image prune -f --external is removing the wrong image.
  2. The test or code is broken somehow when podman is run rootless - note: The rootful test passed fine.
  3. The rootless run_podman 2 rm "$cid" really should be failing but isn't.

I cannot explain why this would pass on Fedora and fail on RHEL :confused: @lsm5 are your Fedora tests running the rootless podman varient? Could the "missing DB" changes have unintentionally altered this behavior? Something else?

cevich avatar Jan 20 '26 16:01 cevich

The remote_root test just finished. Again a minor SELinux failure, and a not ok 362 podman-remote: non-default connection failure. However, I checked the test code and I think this is simply a test bug. The failure happens right at the end, on the final "--remote ps":

...cut...
# [ rc=125 (expected) ]
# # /usr/bin/podman-remote system connection rm defaultconnection
# # /usr/bin/podman-remote system connection rm env-override
# # /usr/bin/podman-remote system connection rm cli-override
# # /usr/bin/podman-remote --remote ps
# CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #| FAIL: exit code is 0; expected 125
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I'm not entirely familiar with this test, but I think perhaps this final check should be removed? It's not clear to me why the default connection should ever fail a ps, but maybe again this has something to do with the "missing DB" handling code (seems unlikely)?

cevich avatar Jan 20 '26 17:01 cevich

remote_rootless test finished. Same sigproxy (ignoring), pasta (ignoring), and (somewhat concerning) podman ps --external failures.


In conclusion, I'm a bit nervous about the ps --external failures. I found some internal results from an automated test run in Sept. 2025 for podman-4.4.1-22.el9_2.4, and the ps --external tests passed.

The only other thing that occurred to me is if maybe this failure is actually caused by a regression or incompatibility with buildah. When I'm running these 9.2 tests, I'm NOT updating the buildah package, so perhaps the run buildah from $IMAGE (at the beginning of the test) isn't behaving as expected?

cevich avatar Jan 20 '26 18:01 cevich

In conclusion, I'm a bit nervous about the ps --external failures. I found some internal results from an automated test run in Sept. 2025 for podman-4.4.1-22.el9_2.4, and the ps --external tests passed.

Is this on 1MT or testing-farm? I have seen it pass consistently on testing-farm RHEL-9.2.0-Nightly compose.

lsm5 avatar Jan 21 '26 09:01 lsm5

pushed another commit. PTAL at the latest.

lsm5 avatar Jan 21 '26 10:01 lsm5

Is this on 1MT or testing-farm? I have seen it pass consistently on testing-farm RHEL-9.2.0-Nightly compose.

This is on 1MT with the podman and podman-* packages built from the HEAD commit on this PR. But, the other container-related bits (like runc and buildah) are stock, whatever is available on 9.2. This is why I'm wondering if maybe it's simply a compatibility problem (i.e. the test makes a call to buildah to setup the container). That could also explain why you're seeing it pass everywhere else.

cevich avatar Jan 21 '26 15:01 cevich

pushed another commit. PTAL at the latest.

Ahh okay, great. I'll run just the local_rootless test and see what difference that makes.

cevich avatar Jan 21 '26 15:01 cevich

Test finished, unfortunately nothing changed, the results are exactly the same. Since this is otherwise passing in Fedora and this PR is growing long in the tooth, let's just ignore this one test failure for now. If it becomes a problem later on in the release process, we can always come back to look deeper and/or fix it. I'm of course open to other opinions as well.

cevich avatar Jan 21 '26 18:01 cevich

@cevich, could you send a full non-snipped test failure(s) that remain on this branch to @mheon and me, please?

I'm not clear if both of the errors a few comments up are still in play, or only the last one.

TomSweeneyRedHat avatar Jan 23 '26 00:01 TomSweeneyRedHat

@TomSweeneyRedHat Sure, my colleague also reproduced the ps --external failure while testing https://github.com/containers/podman/pull/27934 the output is exactly the same as what I was seeing. It's a different podman version, but same testing environment, and the output is exactly the same to my :eye:

podman_ps_external_log.txt Note: This was from a run against 5.4, I've lost the results from v4.4.1, but they look essentially the same.

Also here's the remote_root non-default connection failure (I think this might be a test bug):

non-default_connection_log.txt

cevich avatar Jan 23 '26 14:01 cevich

@cevich it could be Friday afternoon brain, but I'm a little confused by:

Sure, my colleague also reproduced the ps --external failure while testing https://github.com/containers/podman/pull/27934 the output is exactly the same as what I was seeing.

I thought the testing all passed with the updated patch, or are you thinking/saying we need to backport the commits from 27934 into 4.4.1-rhel too?

TomSweeneyRedHat avatar Jan 23 '26 18:01 TomSweeneyRedHat

I thought the testing all passed with the updated patch

The manual testing did not, it failed the same way on "ps --external".

cevich avatar Jan 26 '26 14:01 cevich

@TomSweeneyRedHat Sorry for the confusion, the results I posted here were from Chinmayee's manual testing for RHEL-141490 (podman 5.4). I lost my results from my 4.4.1 testing, but the output is basically the same by my :eye: (sans line numbers). Lokesh said this tests pass in his RHEL testing-farm runs. That's why I think we can ignore it as some odd artifact of how or the environment in which we're manually running the tests.

cevich avatar Jan 26 '26 14:01 cevich