cannot open `/run/user/1000/crun/.../exec.fifo`: No such file or directory

[devurandom]

Describe the bug

Sometimes Podman Compose will be unable to start a pod / Compose environment, mentioning "cannot open /run/user/1000/crun/.../exec.fifo: No such file or directory" and "unable to start container ...: /usr/bin/crun start ... failed: exit status 1" as the error.

I split this out from https://github.com/containers/podman-compose/issues/921, because in that report I encounter two different error messages -- either one or the other.

To Reproduce

Steps to reproduce the behavior: See https://github.com/containers/podman-compose/issues/921#issue-2273448887.

Expected behavior

Pod starts.

Actual behavior

cannot open `/run/user/1000/crun/7eca407523df35a94fa118e422d514ac95b9796ee07cad2b6487e06931510b2e/exec.fifo`: No such file or directory
Error: unable to start container 7eca407523df35a94fa118e422d514ac95b9796ee07cad2b6487e06931510b2e: `/usr/bin/crun start 7eca407523df35a94fa118e422d514ac95b9796ee07cad2b6487e06931510b2e` failed: exit status 1

and then after a while (<60s ususally):

[REDACTED2]    | time="2024-11-08T15:44:42+01:00" level=warning msg="StopSignal SIGTERM failed to stop container [REDACTED3] in 10 seconds, resorting to SIGKILL"

and eventually:

WARNING:podman_compose:container did not shut down after 10 seconds, killing

Note that the "failed to stop container" message does not come right after the "unable to start container" message and not right before the "container did not shut down after 10 seconds" message. Significant (>10s) can pass between each message (while other containers still output their regular log messages).

Also note that the container mentioned in "unable to start container" and the container mentioned in "failed to stop container" are never the same and that the container mentioned as the source of the "failed to stop container" ("[REDACTED2]") is never the same container as the one mentioned in the message ("[REDACTED3]").

Output

❯ grep PRETTY /etc/os-release
PRETTY_NAME="Fedora Linux 41 (KDE Plasma)"

❯ podman-compose --version
podman-compose version 1.2.0
podman version 5.2.5

$ podman-compose up
Error: adding pod to state: name "[REDACTED]" is in use: pod already exists
Error: creating container storage: the container name "[REDACTED1]" is already in use by 7eca407523df35a94fa118e422d514ac95b9796ee07cad2b6487e06931510b2e. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so.
Error: [... similar for other containers in the pod ...]
[REDACTED1]    | cannot open `/run/user/1000/crun/7eca407523df35a94fa118e422d514ac95b9796ee07cad2b6487e06931510b2e/exec.fifo`: No such file or directory
[REDACTED1]    | Error: unable to start container 7eca407523df35a94fa118e422d514ac95b9796ee07cad2b6487e06931510b2e: `/usr/bin/crun start 7eca407523df35a94fa118e422d514ac95b9796ee07cad2b6487e06931510b2e` failed: exit status 1
[... other containers starting ...]
[REDACTED2]    | time="2024-11-08T15:44:42+01:00" level=warning msg="StopSignal SIGTERM failed to stop container [REDACTED3] in 10 seconds, resorting to SIGKILL"
WARNING:podman_compose:container did not shut down after 10 seconds, killing
WARNING:podman_compose:container did not shut down after 10 seconds, killing

Environment:

• OS: Linux (see above)

[Frontesque]

I'd like to bump this issue.

[backend]  | cannot open `/run/user/1000/crun/678da8fa546f94d94c85ce9a0654489a75b368b33b7730d7fa06ae5613ad05fb/exec.fifo`: No such file or directory
[backend]  | Error: unable to start container 678da8fa546f94d94c85ce9a0654489a75b368b33b7730d7fa06ae5613ad05fb: `/usr/bin/crun start 678da8fa546f94d94c85ce9a0654489a75b368b33b7730d7fa06ae5613ad05fb` failed: exit status 1

$ ls -l /run/user/1000/crun/678da8fa546f94d94c85ce9a0654489a75b368b33b7730d7fa06ae5613ad05fb/
total 32
-rwx------. 1 front front 25741 Jan  3 15:48 config.json
-rwx------. 1 front front   916 Jan  3 15:48 status

[daniel-j-h]

I can reproduce this problem, in fact I'm running into it on a fresh Fedora 41 workstation installation

[postgres] | cannot open `/run/user/1000/crun/34d9e2207cb6e13a619aad5d75d9ce3c47758db1cc5a58694cf18f26d1f4afff/exec.fifo`: No such file or directory
[postgres] | Error: unable to start container 34d9e2207cb6e13a619aad5d75d9ce3c47758db1cc5a58694cf18f26d1f4afff: `/usr/bin/crun start 34d9e2207cb6e13a619aad5d75d9ce3c47758db1cc5a58694cf18f26d1f4afff` failed: exit status 1

daniel@fedora /tmp> cat /etc/fedora-release
Fedora release 41 (Forty One)

daniel@fedora /tmp> podman compose --version
podman-compose version 1.2.0
podman version 5.3.1

In my case I have three services: a frontend that depends on a backend and a backend that depends on postgres. The postgres image I'm running is docker.io/library/postgres:17-bookworm and for some reason the postgres container seems to be the one affected from this behavior in my case.

It seems like the behavior is non-deterministic and after cycling up/down a bunch of times I can start the container up just fine.

[az-z]

It seems like the behavior is non-deterministic and after cycling up/down a bunch of times I can start the container up just fine.

oh, yeah. My "favorite pastime" - to get podman compose to actually clean whatever it has to, I have to rerun "down" 2-3 times. It has been like this for at least a year now.

I'm getting this behavior with "depends_on": service_healthy. The depends_on is still not working.

[az-z]

It looks like I ran into this same issue too. Check #1119 .

@devurandom , @daniel-j-h : try adding "version: 3.8" to the top ( see the bottom of my ticket #1119 ).

[devurandom]

It looks like I ran into this same issue too. Check #1119 .

@devurandom , @daniel-j-h : try adding "version: 3.8" to the top ( see the bottom of my ticket #1119 ).

Thanks! I had not set version, because according to https://github.com/compose-spec/compose-spec/blob/main/spec.md#version-top-level-element-obsolete it is obsolete. I'll try setting it again and observe for a while, to see whether this error appears again.

[devurandom]

Even with version: "3.8" I got:

cannot open `/run/user/1000/crun/7be967fb49cc8dc9aa483f0210c0e8f6d73f620735e3af375ad19de4a55bfcf5/exec.fifo`: No such file or directory
Error: unable to start container 7be967fb49cc8dc9aa483f0210c0e8f6d73f620735e3af375ad19de4a55bfcf5: `/usr/bin/crun start 7be967fb49cc8dc9aa483f0210c0e8f6d73f620735e3af375ad19de4a55bfcf5` failed: exit status 1

And with version: "3.20" I still got https://github.com/containers/podman-compose/issues/921.

Both errors might be independent of the version setting -- I did not try often enough to be able to claim that either one does not happen with a certain version.

[az-z]

I consider trying docker-compose plug-in today. If I can't properly orchestrate dependency, it is quite a problem.

https://github.com/containers/podman/pull/19297

On Fri, Jan 24, 2025, 6:24 AM Dennis Schridde @.***> wrote:

Even with version: "3.8" I got:

cannot open /run/user/1000/crun/7be967fb49cc8dc9aa483f0210c0e8f6d73f620735e3af375ad19de4a55bfcf5/exec.fifo: No such file or directory Error: unable to start container 7be967fb49cc8dc9aa483f0210c0e8f6d73f620735e3af375ad19de4a55bfcf5: /usr/bin/crun start 7be967fb49cc8dc9aa483f0210c0e8f6d73f620735e3af375ad19de4a55bfcf5 failed: exit status 1

And with version: "3.20" I still got #921 https://github.com/containers/podman-compose/issues/921.

Both errors might be independent of the version setting -- I did not try often enough to be able to claim that either one does not happen with a certain version.

— Reply to this email directly, view it on GitHub https://github.com/containers/podman-compose/issues/1072#issuecomment-2612291488, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGHHXMCYRZUXEQUFGLKDCD2MIPIJAVCNFSM6AAAAABRNVVVR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJSGI4TCNBYHA . You are receiving this because you commented.Message ID: @.***>

[az-z]

ah! I recalled that doing the MDM routine usually helped me before.

The Magical Dance Moves (MDM):

1. podman-compose .. down
2. podman-compose ... down
3. whistle 4 seconds
4. podman-compose up

and life is better.

I wish I were joking.