snapshots: Fix error when switching different snapshotters

[ChengyuZhu6]

On proxy driver, we found an issue that if we use other snapshotter (like overlayfs) to pull an image and then switching to nydus to pull the same image, we would get the error: "error unpacking image: failed to extract layer sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68: failed to get reader from content store: content digest sha256:fbe1a72f5dcd08ba4ca3ce3468c742786c1f6578c1f6bb401be1c4620d6ff705: not found" same issue: https://github.com/kata-containers/kata-containers/issues/8407#:~:text=edited-,Issue%201,-error%20unpacking%20image

This error occurs because the image puller has an optimization that skips downloading the layer related to the image, if it is already unpacked as a snapshot in the default snapshotter (overlayfs).

The error could be fixed on proxy driver. The logic behind proxy mode is simple: by inserting the label containerd.io/snapshot/nydus-proxy-mode=true into the snapshot labels for ro layers, and for the rw layer, we call mountWithKataVolume to set up mounts with proxy volume.

Therefore, the refined logic would be as follows on proxy driver: Verify whether the driver is proxy and the snapshot labels to ensure the inclusion of containerd.io/snapshot/nydus-proxy-mode=true. If this condition is met, proceed as usual; if not, return an error. Subsequently, prompt containerd to reprepare the snapshots for the image, even if the image's snapshots have been previously prepared by another snapshotter.

[ChengyuZhu6]

I think the method maybe not perfect, but it could resolve the current issue.