stepchowfun
Podman regression in v0.25.1
Description
I know that podman is not officially supported, but it currently seems to work fine. However, the latest release seems to contain a regression.
Instructions to reproduce the bug
- Run docuum on rootless podman with
$ podman run --init --rm --tty --name docuum --mount type=bind,src=/run/user/1101/podman/podman.sock,dst=/var/run/docker.sock --mount type=bind,source=/home/runner/docuum,target=/root stephanmisc/docuum --threshold '10 GB' --keep 'docker.io/stephanmisc/docuum' - get the following log output:
[2025-11-18 18:31:59 +00:00 INFO] Performing an initial vacuum on startup…
Error response from daemon: unknown container state: restarting: invalid argument
[2025-11-18 18:32:00 +00:00 ERROR] Unable to list containers.
[2025-11-18 18:32:00 +00:00 INFO] Retrying in 5 seconds…
...
[2025-11-18 18:34:28 +00:00 INFO] Performing an initial vacuum on startup…
Error response from daemon: unknown container state: dead: invalid argument
[2025-11-18 18:34:29 +00:00 ERROR] Unable to list containers.
[2025-11-18 18:34:29 +00:00 INFO] Retrying in 5 seconds…
...
- Try with
$ podman run --init --rm --tty --name docuum --mount type=bind,src=/run/user/1101/podman/podman.sock,dst=/var/run/docker.sock --mount type=bind,source=/home/runner/docuum,target=/root stephanmisc/docuum:0.25.0 --threshold '10 GB' --keep 'docker.io/stephanmisc/docuum' - observe that it works
It would be really great if you could look into this to get podman working again. Thank you very much!
Ah, looks like that was from https://github.com/stepchowfun/docuum/pull/341, intended to fix https://github.com/stepchowfun/docuum/issues/237 (workaround for a Docker bug). We could revert that PR, but it would bring back the original Docker issue.
If Podman aims to be compatible with Docker, I wonder if this could be fixed upstream there. Or maybe we could fix the original Docker issue that led to that PR.
It seems like "restarting" is not a valid podman state. I can try to ask upstream, but I wouldn't know if it would be better to simply ignore this or explicitly error.
I am happy to ask podman to improve compatibility here. But it really feels like an xy-problem. If the problem is that docker containers can get stuck and you exclude one state, I feel like technically there is no guarantee that there won't be changes in states or other stuck containers where inspect would fail that suffer from the same issue. My naive suggestion to fix the problem is ignoring the "container doesn't exist" error from docker in case inspection fails.
Please let me know if I should proceed with discussing with podman anyway.
Another potential hotfix on the docuum side could be to not filter by status=restarting as I would assume this container state is not super interesting and should only be temporary. But I don't know much about how docuum works and if this is safe enough.
My naive suggestion to fix the problem is ignoring the "container doesn't exist" error from docker in case inspection fails.
I agree with this! The only complication is that Docuum queries Docker about a whole batch of containers at once rather than querying them individually. So if the whole batch fails due to one container, we'd need to add some retry logic that splits up the problematic batch or attempts each container individually. It seems like a relatively small amount of work—I'd be happy to review a PR for it.
Another potential hotfix on the docuum side could be to not filter by status=restarting as I would assume this container state is not super interesting and should only be temporary.
From your logs it seems like dead is also an invalid status for Podman:
Error response from daemon: unknown container state: dead: invalid argument
So we'd need to account for all of these (is Docker's problematic removing status even valid in Podman?).
Additional commentary on previous message:
add some retry logic that splits up the problematic batch or attempts each container individually.
Ideally we'd do the former (e.g., splitting the failing batch into halves). Querying each container individually would be a bummer for performance (the batching optimization was important).