qm container taking long time to start on mac

[jtligon]

created qcow2 on mac. running on macos 26.1

Client:        Podman Engine
Version:       5.6.2
API Version:   5.6.2
Go Version:    go1.25.1
Built:         Tue Sep 30 10:50:46 2025
Build Origin:  brew
OS/Arch:       darwin/arm64

Server:       Podman Engine
Version:      5.6.2
API Version:  5.6.2
Go Version:   go1.24.7
Git Commit:   9dd5e1ed33830612bc200d7a13db00af6ab865a4
Built:        Mon Sep 29 20:00:00 2025
OS/Arch:      linux/arm64

qemu: QEMU emulator version 10.1.2

used http://gitlab.com/CentOS/automotive/autosd-image-score

ran qcow2 using auto-image-runner --nographics ...

CONTAINER ID  IMAGE       COMMAND     CREATED        STATUS                   PORTS       NAMES
c639e0053cdc              /sbin/init  9 minutes ago  Up 9 minutes (starting)              qm

verified with another manifest.

[Yarboa]

Thanks for reporting Can you elaborate more on taking long time ? does it complete success? Can you please attach output of systemctl status qm

What about journactl -u qm? Are there any AVCs? Can you try do set selinux to permissive?

[jtligon]

Yep. It's not leaving the starting state. Tried 2 different images and qm never finishes starting

[Yarboa]

Yep. It's not leaving the starting state. Tried 2 different images and qm never finishes starting

Yes, i got it, update the comment above

[jtligon]

nothing suspicious that I see

Oct 02 00:00:08 esdv-score-main-compute podman[336]: 2025-10-02 00:00:08.571080045 +0000 UTC m=+0.142292709 sy
stem refresh
Oct 02 00:00:08 esdv-score-main-compute podman[336]: 2025-10-02 00:00:08.59941542 +0000 UTC m=+0.170628084 con
tainer create 4d8e80a2d443dfc34f9f35b02c3782c9f60b5228f79bd6efdf71ee8171eecaf2 (image=, name=qm, PODMAN_SYSTEM
D_UNIT=qm.service)
Oct 02 00:00:08 esdv-score-main-compute podman[336]: 2025-10-02 00:00:08.84097442 +0000 UTC m=+0.412187126 con
tainer init 4d8e80a2d443dfc34f9f35b02c3782c9f60b5228f79bd6efdf71ee8171eecaf2 (image=, name=qm, PODMAN_SYSTEMD_
UNIT=qm.service)
Oct 02 00:00:08 esdv-score-main-compute podman[336]: 2025-10-02 00:00:08.843129629 +0000 UTC m=+0.414342334 co
ntainer start 4d8e80a2d443dfc34f9f35b02c3782c9f60b5228f79bd6efdf71ee8171eecaf2 (image=, name=qm, PODMAN_SYSTEM
D_UNIT=qm.service)
Oct 02 00:00:08 esdv-score-main-compute qm[336]: 4d8e80a2d443dfc34f9f35b02c3782c9f60b5228f79bd6efdf71ee8171eec
af2
Oct 02 00:00:08 esdv-score-main-compute systemd[1]: Started qm.service.
Oct 02 00:01:12 esdv-score-main-compute podman[635]: 2025-10-02 00:01:12.366230576 +0000 UTC m=+0.013643376 co
ntainer died 4d8e80a2d443dfc34f9f35b02c3782c9f60b5228f79bd6efdf71ee8171eecaf2 (image=, name=qm, PODMAN_SYSTEMD
_UNIT=qm.service)
Oct 02 00:01:12 esdv-score-main-compute podman[635]: 2025-10-02 00:01:12.575205492 +0000 UTC m=+0.222618251 co
ntainer remove 4d8e80a2d443dfc34f9f35b02c3782c9f60b5228f79bd6efdf71ee8171eecaf2 (image=, name=qm, PODMAN_SYSTE
MD_UNIT=qm.service)
Oct 02 00:01:12 esdv-score-main-compute systemd[1]: qm.service: Deactivated successfully.
Oct 02 00:01:12 esdv-score-main-compute systemd[1]: qm.service: Consumed 936ms CPU time, 119.2M memory peak.
-- Boot 788593a9b14c4273a8c79455e2e7e699 --
Oct 02 00:00:10 esdv-score-main-compute systemd[1]: Starting qm.service...
Oct 02 00:00:10 esdv-score-main-compute podman[315]: 2025-10-02 00:00:10.907308963 +0000 UTC m=+0.132581959 sy
stem refresh
Oct 02 00:00:10 esdv-score-main-compute podman[315]: 2025-10-02 00:00:10.927855088 +0000 UTC m=+0.153128084 co
ntainer create 5947ee647bff9e3ccd3d4f3ad89ed85a6f649d18d42a56392baa8578d3ee5e33 (image=, name=qm, PODMAN_SYSTE
MD_UNIT=qm.service)
Oct 02 00:00:11 esdv-score-main-compute podman[315]: 2025-10-02 00:00:11.169576546 +0000 UTC m=+0.394849584 co
ntainer init 5947ee647bff9e3ccd3d4f3ad89ed85a6f649d18d42a56392baa8578d3ee5e33 (image=, name=qm, PODMAN_SYSTEMD
_UNIT=qm.service)
Oct 02 00:00:11 esdv-score-main-compute podman[315]: 2025-10-02 00:00:11.174761421 +0000 UTC m=+0.400034459 co
ntainer start 5947ee647bff9e3ccd3d4f3ad89ed85a6f649d18d42a56392baa8578d3ee5e33 (image=, name=qm, PODMAN_SYSTEM
D_UNIT=qm.service)
Oct 02 00:00:11 esdv-score-main-compute qm[315]: 5947ee647bff9e3ccd3d4f3ad89ed85a6f649d18d42a56392baa8578d3ee5
e33
Oct 02 00:00:11 esdv-score-main-compute systemd[1]: Started qm.service.```

[Yarboa]

Is it c10s based? what is the command you used for buidling the image?

I see the same issue here

cat /etc/os-release 
NAME="Red Hat In-Vehicle Operating System"
VERSION="2.0"
ID="rhivos"
ID_LIKE="centos fedora rhel"
VERSION_ID="2.0"

journalctl -xeu qm.service
Oct 02 09:11:46 localhost qm[4729]: time="2025-10-02T09:11:46Z" level=warning msg="The storage 'driver' option should be>
Oct 02 09:11:46 localhost qm[4729]: Error: statfs /var/qm/tmp: no such file or directory
Oct 02 09:11:46 localhost systemd[1]: qm.service: Main process exited, code=exited, status=125/n/a

I had to create /var/qm/tmp, since it was not visible i assume due to overlayfs I see tmp dir here, and this is why qm is not starting, it is generic issue not only on mac

ls -ltra /usr/lib/qm/rootfs/tmp/
total 8
drwxrwxrwt.  2 root root 4096 Apr  2  2025 .
dr-xr-xr-x. 18 root root 4096 Nov 13  2025 ..

and there are wonderful AVCS :)

time->Thu Oct  2 00:08:10 2025
type=PROCTITLE msg=audit(1759363690.531:215): proctitle="/sbin/init"
type=SYSCALL msg=audit(1759363690.531:215): arch=c00000b7 syscall=34 success=no exit=-13 a0=ffffffffffffff9c a1=aaaadb73ace0 a2=1c0 a3=0 items=0 ppid=1093 pid=1095 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd" exe="/usr/lib/systemd/systemd" subj=system_u:system_r:qm_t:s0 key=(null)
type=AVC msg=audit(1759363690.531:215): avc:  denied  { write } for  pid=1095 comm="systemd" name="tmp" dev="sdf20" ino=12 scontext=system_u:system_r:qm_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=dir permissive=0
[root@localhost ~]#

[jtligon]

AIB_SH_URL := https://gitlab.com/CentOS/automotive/src/automotive-image-builder/-/raw/main/auto-image-builder.sh?ref_type=heads AIB_DISTRO := autosd10 AIB_MODE := package AIB_TARGET = qemu AIB_EXPORT = qcow2 AIB_IMAGE := image AIB_OUTPUT := disk.qcow2

[dougsland]

I am going to try this one, first question I have is: qm version? We just recently released qm 1.0

[jtligon]

Which version was pulling and building last week?

[dougsland]

Which version was pulling and building last week?

So it suppose to be 1.0 in your env.

[dougsland]

FYI: I tried to reproduce this on CentOS 10 (Linux) using different Podman versions, but I couldn’t. QM started normally, and the nested containers worked as expected.

# cat /etc/redhat-release
CentOS Stream release 10 (Coughlan)

# rpm -qa | grep qm
qm-1.0-1.el10_2.noarch

# podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED       STATUS       PORTS       NAMES
77c0aff6f381              /sbin/init  1 second ago  Up 1 second              qm

# rpm -qa | grep -i podman
podman-5.4.0-3.el10.aarch64

# rpm -qa | grep podman
podman-5.6.0-2.el10.aarch64

[dougsland]

Hey Jeff,

I tried to reproduce this locally under several scenarios, but I wasn’t able to (on Linux). On macOS, I still need to understand exactly how you're enable the environment so I will give a shot as well.

When you mention QEMU, are you referring to: Image generated by running Linux on QEMU (manually triggered) and running the image via auto-image-running in the same env. If you have a step-by-step sequence of what you’re doing, that would be very helpful.

Finally, when this happens, are you able to triggered others containers without this delay ?

[Yarboa]

I looked again in c10s image It seems that container is up and running Looking at container inspect, the following is seen

               "Health": {
                    "Status": "starting",
                    "FailingStreak": 0,
                    "Log": null
               },

rpm -q podman -q qm
podman-5.6.0-6.el10_1.x86_64
qm-1.0-1.el10iv.noarch

still digging into that,

I assume it is related to podman and quadlets sine some how the (Starting) or (Healthy) are part of health check, compare to regular container run

I have added that cat /etc/containers/systemd/qm.container.d/tmp.conf

[Container]
HealthCmd=/usr/bin/systemctl is-system-running | grep -q 'running' || exit 1
# Wait 5 minutes before running the first check (allows systemd to fully boot).
HealthStartPeriod=5s
# Run the check every 30 seconds.
HealthInterval=5s

And container is healthy

Let me check the default command created by podman quadlet generator

Interesting adding a service for the ubi image

cat /etc/containers/systemd/app.container
[Install]
WantedBy=default.target

[Container]
Image=ubi9
Exec=/sbin/init
ContainerName=app

[Service]
Restart=always

No health check status

podman ps
CONTAINER ID  IMAGE                                   COMMAND     CREATED        STATUS        PORTS       NAMES
2d7bc6613d6f  registry.access.redhat.com/ubi9:latest  /sbin/init  2 seconds ago  Up 2 seconds              app
[root@localhost ~]# cat /etc/containers/systemd/app.container

[dougsland]

I was able to reproduce as well (Linux). The container is running just fine but the status is starting forever.

# cat /etc/redhat-release
Red Hat In-Vehicle Operating System release 2.0

# rpm -qa | grep -i podman
python3-podman-5.6.0-1.el10.noarch
podman-5.6.0-6.el10_1.x86_64
cockpit-podman-116-1.el10.noarch
podman-docker-5.6.0-6.el10_1.noarch

# podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED      STATUS                 PORTS       NAMES
c4c0e5372bef              /sbin/init  1 hours ago  Up 1 hours (starting)              qm

No healthcheck:

 podman inspect qm --format '{{json .Config.Healthcheck}}'
{}

The state health:

podman inspect qm --format '{{json .State.Health}}'
{"Status":"starting","FailingStreak":0,"Log":null}

The state status:

# podman inspect qm --format '{{.State.Status}}'
running

[dougsland]

Discussed with podman engineering team and opened the following issue: https://github.com/containers/podman/issues/27651