composefs: committing the finished image: failed to put layer using a partial pull: rename xxx/diff: no such file or directory

[edsantiago]

The gist seems to be:

# podman-remote [something that does a pull]
Error: committing the finished image: \
    failed to put layer using a partial pull: \
    rename /var/lib/containers/storage/overlay/staging/1464572085 \
        /var/lib/containers/storage/overlay/SHA/diff: \
    no such file or directory

Seems to be remote-only.

Been trying all afternoon to reproduce this. I can't. So here are links to failure logs:

• f39, in podman load from URL test
• rawhide, ibid
• f39 aarch64, in pod something infra something test

[edsantiago]

f39 rootless:

[+0817s] not ok 356 [200] podman pod create should fail when infra-name is already in use
...
<+195ms> # $ podman-remote --url unix:///tmp/podman_tmp_ktBx --noout pod create --name nifusgth72 --infra-name infra_container_z257ixa6bk --infra-image quay.io/libpod/k8s-pause:3.5
<+1.29s> # Error: failed to make pod: committing the finished image: failed to put layer using a partial pull: rename /home/some16421dude/.local/share/containers/storage/overlay/staging/2329401209 /home/some16421dude/.local/share/containers/storage/overlay/dee215ffc666313e1381d3e6e4299a4455503735b8df31c3fa161d2df50860a8/diff: no such file or directory

f39 root:

[+0792s] not ok 279 [120] podman load - from URL
...
<+055ms> # # podman-remote --url unix:///tmp/podman_tmp_pBmm run -d --name myweb -p 5340:80 -v /tmp/podman_bats.xc2Uar/myimage-nlhjn08D.tar:/var/www/image.tar:Z -w /var/www quay.io/libpod/testimage:20240123 /bin/busybox-extras httpd -f -p 80
<+1.98s> # Trying to pull quay.io/libpod/testimage:20240123...
         # Getting image source signatures
         # Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
         # Copying blob sha256:abbb4fed5b1fdc439f1f0138b257ea2ae6a69ddf7d9508eaf6ab268636b0a842
         # Copying config sha256:1f6acd4c4a1d4d39395870e1c12dbb68da27dd432afecf86e7cb6e23cf4b75d0
         # Writing manifest to image destination
         # Error: committing the finished image: failed to put layer using a partial pull: rename /var/lib/containers/storage/overlay/staging/3873943774 /var/lib/containers/storage/overlay/25a964f60e84b2d24f8098376c985a5c057bea74d699ee50c93e417c26516cc3/diff: no such file or directory

[giuseppe]

we should not be so aggressive to cleanup the staging directory: https://github.com/containers/storage/blob/3cbd3c6fcd26cc961c1b462575aa1f0c597a24b1/drivers/overlay/overlay.go#L883

I am not sure yet how to address it, I'll keep working on this

[giuseppe]

opened a PR:

• https://github.com/containers/storage/pull/1916