Older Toolbx Containers fail to run with crun >= 1.8.2

[DaliborKr]

Summary

I encountered an error while testing the backward compatibility of the Toolbx containers. Running older Toolbx containers (created with Toolbx version 0.0.96) fails with newer versions of crun (starting from version 1.8.2).

Error

Error: crun: chown `/dev/pts/N`: Operation not permitted: OCI permission denied

Steps to Reproduce

Reproduced using Podman on Fedora 39 Workstation with Podman version 4.7.0 to ensure compatibility with older crun versions.

1. Create a container:

   podman create --name toolbox-container-old --userns=keep-id --user root:root  --volume /dev:/dev:rslave registry.fedoraproject.org/fedora:42 sleep 5m
   

2. Start the container:

   podman start toolbox-container-old
   

3. Attempt to execute a command:

   podman exec --tty --user 1000 toolbox-container-old pwd
   

• This results in the mentioned error when using crun 1.8.2 (crun-1.8.2-2.fc39.x86_64.rpm).
• However, this sequence works as expected with crun 1.8.1 (crun-1.8.1-1.fc39.x86_64.rpm).

System Information

• OS: Fedora 39 Workstation (Fedora 39 ISO)
• Podman version: 4.7.0 (4.7.0-1.fc39.x86_64)
• crun versions tested:
  • 1.8.1 (1.8.1-1.fc39.x86_64) -> works
  • 1.8.2 (1.8.2-2.fc39.x86_64) -> fails

I used Fedora 39 Workstation to ensure having a version of podman old enough, compatible with older versions of crun.

Related Discussions

This regression appears to be related to containers/crun#1159. The issue was previously discussed in containers/crun#1158:

Is it possible that podman exec --tty --user $UID ... stopped working due to #1159, unless the container was created with --mount type=devpts,destination=/dev/pts? Is that expected?

I was trying not have the separate devpts to see if it helps with containers/toolbox#568

Background

Newer versions of Toolbx now use the following podman create option to avoid this issue:

--mount type=devpts,destination=/dev/pts

However, containers created with older versions (such as Toolbx 0.0.96) did not include this mount, resulting in this failure with newer crun versions.

[giuseppe]

Fedora 39 is EOL so whatever fix we could do in crun won't get there, and even the Toolbox container itself should not be based on some old distro that doesn't get any update.

Can't the toolbox container just be recreated?

[debarshiray]

Fedora 39 is EOL so whatever fix we could do in crun won't get there, and even the Toolbox container itself should not be based on some old distro that doesn't get any update.

Fedora 39 is a bit of a red herring because it was only used to bisect the problem to the exact crun version and change, because as we bisect deeper and deeper into crun, a more modern Podman stack (like the one on Fedora 42) stops working with it.

The problem that @DaliborKr found also occurs on Fedora 42 because it has crun >= 1.8.2.

If the regression is fundamentally impossible to fix in crun, then I guess we will mark those containers as deprecated or unsupported.

[giuseppe]

if anyone cares to open a PR, I'll be glad to review it