[TRACKER ONLY]: (qm-minimal.mpp.yml) 0.6.2: nested container: overlay: can't stat imageStore dir /usr/share/containers/storage: stat /usr/share/containers/storage: no such file or directory

[dougsland]

Just found this one: nested container: overlay: can't stat imageStore dir /usr/share/containers/storage: stat /usr/share/containers/storage: no such file or directory

See-Also: #405

git clone https://gitlab.com/CentOS/automotive/sample-images.git
pushd osbuild-manifests/
    make cs9-qemu-qm-minimal-regular.x86_64.qcow2
    ./runvm --nographics ./cs9-qemu-qm-minimal-regular.x86_64.qcow2

    user: root pass: password

    [root@localhost ~]# podman exec -it qm bash
    bash-5.1#  podman run -it --name fedora fedora  /bin/bash
    Error: overlay: can't stat imageStore dir /usr/share/containers/storage: stat /usr/share/containers/storage: no such file 
    or directory 

Another test:

bash-5.1# podman ps
Error: overlay: can't stat imageStore dir /usr/share/containers/storage: stat /usr/share/containers/storage: no such file or directory

[dougsland]

@Yarboa for some reason, I thought this one was fixed via https://github.com/containers/qm/blob/main/setup#L91-L116

Digging it.

[dougsland]

talked with @rhatdan and his debug approach helped to identify the issue and he mentioned that RHEL9.5 should be fixed by default.

Debugging:

bash-5.1#  podman --root /usr/share/containers/storage pull alpine
Error: creating runtime static files directory "/usr/share/containers/storage/libpod": mkdir /usr/share/containers/storage: read-only file system

The final solution for this error is (in the host, not in QM):

mkdir /usr/lib/qm/rootfs/usr/share/containers/storage
mkdir /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-images
touch /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-images/images.lock
mkdir /usr/lib/qm/rootfs/usr/share/containers/storage/libpod
mkdir /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-layers
touch /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-layers/layers.lock

Now podman is working as expected

bash-5.1# podman pull alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Writing manifest to image destination
05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd

bash-5.1# podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

bash-5.1# podman images
REPOSITORY                TAG         IMAGE ID      CREATED       SIZE
docker.io/library/alpine  latest      05455a08881e  3 months ago  7.67 MB

bash-5.1# podman info
host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: d807bb8c1de3dc05fb66c77d2979a7f6903804bf'
  cpuUtilization:
    idlePercent: 99.19
    systemPercent: 0.54
    userPercent: 0.27
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: centos
    version: "9"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.14.0-438.391.el9iv.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1648013312
  memTotal: 2060533760
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: Unknown
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 0h 7m 15.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/share/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 7938207744
  graphRootUsed: 1022078976
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: true
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.9.4-dev
  Built: 1710930166
  BuiltTime: Wed Mar 20 10:22:46 2024
  GitCommit: ""
  GoVersion: go1.21.7 (Red Hat 1.21.7-1.el9)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-dev

bash-5.1#

Now I need to figure out our code and see why what we have is not working and prepare a patch.

[Yarboa]

@dougsland it is all described in the Spike work setup script is not running while building autosd image (regular/ostree)

in qm setup script, it update the qm:podman cofn files with the relevant data. this setup is not running inside osbuild

So there are two option to bring it into work

1. as setup script update config files
2. as suggested above.

Whatever chosen, should work both qm deployments

Anyway option 2, requires minor fix in qm.mpp

[dougsland]

@pbrilla-rh This issue might not be a blocker for the release as this behavior only happens in the images/qm-minimal.mpp.yml. If QM minimal is NOT a requirement for the release, we keep this bug opened and move forward with the release.

I have tested the image cs9-qemu-qmcontainer-regular.x86_64.qcow2 (images/qmcontainer.mpp.yml) and it WORKED just fine with podman + nested containers in QM. However, I see another issue with the OSTREE image version: cs9-qemu-qmcontainer-ostree.x86_64.qcow2 (not really related to QM code but still):

# make cs9-qemu-qmcontainer-ostree.x86_64.qcow2
osbuild-mpp -I . -D image_type="\"ostree\"" -D arch=\"x86_64\" -D distro_name="\"cs9\"" -D target="\"qemu\""    images/qmcontainer.mpp.yml _build/cs9-qemu-qmcontainer-ostree.x86_64.json
osbuild --checkpoint build --store _build/osbuild_store --output-directory _build/image_output --cache-max-size=1GB --export qcow2 _build/cs9-qemu-qmcontainer-ostree.x86_64.json
_build/cs9-qemu-qmcontainer-ostree.x86_64.json has errors:

.pipelines[6].stages[5]:
  could not find schema information for 'org.osbuild-auto.ostree.pre-gen'

make: *** [Makefile:171: cs9-qemu-qmcontainer-ostree.x86_64.qcow2] Error 2

Is this image requirement for the release? I guess not, as the others releases always went ahead without it... but worth double check right?

I was asking myself why qmcontainer.mpp.yml worked and qm-minimal.mpp.yml not?
What's the difference to make it work?

+  - type: org.osbuild.mkdir
+    options:
+      paths:
+      - path: /usr/share/containers/storage/overlay-images
+        parents: true
+      - path: /usr/share/containers/storage/overlay-layers
+        parents: true
+  - type: org.osbuild.copy
+    inputs:
+      inlinefile:
+        type: org.osbuild.files
+        origin: org.osbuild.source
+        mpp-embed:
+          id: empty
+          path: ../files/empty
+    options:
+      paths:
+      - from:
+          mpp-format-string: input://inlinefile/{embedded['empty']}
+        to: tree:///usr/share/containers/storage/overlay-images/images.lock
+      - from:
+          mpp-format-string: input://inlinefile/{embedded['empty']}
+        to: tree:///usr/share/containers/storage/overlay-layers/layers.lock

Even applying this directly in the qm-mininal yaml file still not working and might explain why this change was only added in qmcontainer.mpp.yml and not in the minimal version in the time. When adding In the qm-minimal during the build it complains about readonly file system during the lock files creation (even few lines above creating the dirs). At least in my test environment.

I also would like to suggest a little change to the solution, move the code to image.ipp.yml (to deliver the same behavior in ALL qm images in a single place) instead of dup the code to qm-minimal until the podman 5 lands in EL9. However, it's also not working for the same reason I mentioned above (the readonly in the lock files). My question for now, should we move this one to the next release pending tasks?

Finally, thanks @Yarboa for always pinging me in the slack offering help.

diff --git a/osbuild-manifests/include/image.ipp.yml b/osbuild-manifests/include/image.ipp.yml
index 4f7d9aa..32872bb 100644
--- a/osbuild-manifests/include/image.ipp.yml
+++ b/osbuild-manifests/include/image.ipp.yml
@@ -125,6 +125,30 @@ mpp-vars:
                 parents: true
               - path: /var/qm/
                 parents: true
+              # START: workaround until podman 5 is included into the distro
+              # https://github.com/containers/qm/issues/406
+              - path: /usr/lib/qm/usr/share/containers/storage
+                parents: true
+              - path: /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-images
+                parents: true
+              - path: /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-layers
+                parents: true
+              # END: workaround until podman 5 is included into the distro
+              # https://github.com/containers/qm/issues/406
+
+            # START: workaround until podman 5 is included into the distro
+            # https://github.com/containers/qm/issues/406
+            type: org.osbuild.truncate
+            options:
+               filename: /usr/lib/qm/usr/share/containers/storage/overlay-images/images.lock
+               size: '0'
+            type: org.osbuild.truncate
+            options:
+               filename: /usr/lib/qm/usr/share/containers/storage/overlay-layers/layers.lock
+               size: '0'
+            # END: workaround until podman 5 is included into the distro
+            # https://github.com/containers/qm/issues/406

[Yarboa]

@dougsland we should update build doc, just install sudo dnf install -y osbuild-auto

This issue should be managed in the sig, not here

[dougsland]

Just confirmed, https://gitlab.com/CentOS/automotive/sample-images/-/merge_requests/506 worked for me (qm-minimal). Probably there are others ways to fix it (like installing osbuild-auto + the patch I mentioned above but let's move forward with the simplest as possible). +1 @Yarboa Thanks! Waiting the review and merge to close this one.

This issue is just a tracker at this moment.

[dougsland]

Nothing related to QM, @Yarboa is taking care of it in the sample-image project.