cannot run podman from root user

[paleozogt]

Issue Description

I am not able to run containers within an EKS pod:

# whoami
root

# podman run --rm -it ubuntu ls
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning. 
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning. 
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup for cpuset: mkdir /sys/fs/cgroup/cpuset/libpod_parent: read-only file system 
Error: crun: creating cgroup directory `/sys/fs/cgroup/hugetlb/libpod_parent/libpod-791d15046ae0dfe30312327f956ab5ada9012fc54f743318d23058c9bf5dc019`: No such file or directory: OCI runtime attempted to invoke a command that was not found

This seems to have started sometime around v4.9.4. In v4.6.2 it works:

# podman version
Client:       Podman Engine
Version:      4.6.2
API Version:  4.6.2
Go Version:   go1.19.12
Built:        Mon Aug 28 19:38:31 2023
OS/Arch:      linux/amd64

# whoami
root

# podman run --rm -it ubuntu ls
bin  boot  dev	etc  home  lib	lib64  media  mnt  opt	proc  root  run  sbin  srv  sys  tmp  usr  var

Steps to reproduce the issue

Steps to reproduce the issue

1. Run a pod using quay.io/buildah/stable:v1.37.0
2. Install podman in the pod with yum install -y podman
3. Try to run a container, e.g., podman run --rm -it ubuntu ls

Describe the results you received

WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup for cpuset: mkdir /sys/fs/cgroup/cpuset/libpod_parent: read-only file system 
Error: crun: creating cgroup directory `/sys/fs/cgroup/hugetlb/libpod_parent/libpod-791d15046ae0dfe30312327f956ab5ada9012fc54f743318d23058c9bf5dc019`: No such file or directory: OCI runtime attempted to invoke a command that was not found

Describe the results you expected

Be able to run the container.

podman info output

podman info

host:
  arch: amd64
  buildahVersion: 1.37.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 93.25
    systemPercent: 3.95
    userPercent: 2.81
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "40"
  eventLogger: file
  freeLocks: 2048
  hostname: foobar-37
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.10.220-209.869.amzn2.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 10196676608
  memTotal: 16368697344
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.1-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240814.g61c0b0d-1.fc40.x86_64
    version: |
      pasta 0^20240814.g61c0b0d-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 17h 24m 25.00s (Approximately 0.71 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-1.fc40.x86_64
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 483171217408
  graphRootUsed: 7751495680
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.2.1
  Built: 1723593600
  BuiltTime: Wed Aug 14 00:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.5
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.1

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

Running in AWS EKS pods, which still use cgroups v1.

Additional information

Curiously, back in v4.6.2, it defaults to running rootless even when run from the root user:

# podman version
Client:       Podman Engine
Version:      4.6.2
API Version:  4.6.2
Go Version:   go1.19.12
Built:        Mon Aug 28 19:38:31 2023
OS/Arch:      linux/amd64

# whoami
root

# podman info | grep rootless
    rootless: true

While in the current version it defaults to rootful from the root user:

# podman version
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning. 
Client:       Podman Engine
Version:      5.2.1
API Version:  5.2.1
Go Version:   go1.22.5
Built:        Wed Aug 14 00:00:00 2024
OS/Arch:      linux/amd64

# whoami
root

# podman info | grep rootless
...
    rootless: false

... which makes sense in a way (since we're running as root after all).

However, when we can't/won't run as rootful, how can we tell podman to run rootless when its run from the root user?