Matt Martin

Per @haircommander's comment on slack, here is where I tried to pull an image with podman. It fails in a similar way: ``` podman pull redacted.com/redacted/redacted-image:redacted-tag Trying to pull redacted.com/redacted/redacted-image:redacted-tag......

> also, what version of podman? 3.4.2

> But I'm unable to reproduce this issue exactly, one reason could be that version v1.32.6 is being used by crio v1.20.6 but no podman was released using this version...

> it could also depend on that particular image. > Is there any way you could share it with us? @giuseppe I'm pretty sure we are seeing this with multiple...

@flouthoc, I'll update our incident response to tell people to try this the next time it happens on a non production node. Hopefully they read the docs. :) Doing this...

@flouthoc, I tried cleaning as you suggested and the image pull still fails with crictl and with podman: ``` # podman --version podman version 3.4.2 # podman system prune --all...

@matthias50 Ugh. And of course I did the `crictl pull` first. When this happens again, I'll try to catch it and perform just the `podman pull` (like I wrote in...

@flouthoc. This happened again and this time we tried the correct sequence of `podman system prune` followed by `podman pull` This time `podman system prune` didn't do anything (probably because...

@haircommander, @flouthoc, we have recently updated to crio 1.22.5. Does this crio have the updated `c/storage` which should not have this issue? If not, which version does?

@flouthoc, it does seem similar. iirc, when I created the directory, it fixed one error but then there were more and I couldn't find a way to reliably fix nodes...