Add --cache-first or --cache-only flag

[rhpvorderman]

Version of Singularity: 3.5.3

Expected behavior

$ singularity exec --containall docker://quay.io/biocontainers/fastqc@sha256:319b8d4eca0fc0367d192941f221f7fcd29a6b96996c63cbf8931dbb66e53348 fastqc --version
FastQC v0.11.9

This will pull the image at the correct sha256 hash.

If you disable the internet, singularity should know this image is already in the cache.

Actual behavior

$ singularity exec --containall docker://quay.io/biocontainers/fastqc@sha256:319b8d4eca0fc0367d192941f221f7fcd29a6b96996c63cbf8931dbb66e53348 fastqc --version
FATAL:   Unable to handle docker://quay.io/biocontainers/fastqc@sha256:319b8d4eca0fc0367d192941f221f7fcd29a6b96996c63cbf8931dbb66e53348 uri: failed to get SHA of docker://quay.io/biocontainers/fastqc@sha256:319b8d4eca0fc0367d192941f221f7fcd29a6b96996c63cbf8931dbb66e53348: pinging docker registry returned: Get https://quay.io/v2/: dial tcp: lookup quay.io on [::1]:53: read udp [::1]:54893->[::1]:53: read: connection refused

Singularity will always check the image sha256 digest online. Even if the sha256 digest is explicitly given.

Feature request.

Add a --cache-first or --cache-only flag to either check the cache first (instead of checking the internet first) or only check the cache. This is useful for HPC environments where the execution nodes do not always have internet access. This way the image can be pulled to the cache first before the job is submitted to the execution node.

[dtrudg]

Though the cache has been restructured for 3.6 to address concurrency issues, it's not trivial for us to do this type of lookup yet.

While this quay.io docker image has an associated hash 319b8d4eca0fc0367d192941f221f7fcd29a6b96996c63cbf8931dbb66e53348 that is not the hash on anything that we put into our cache.

We are retrieving and caching the blobs for that image - but we don't have an easy association in place from the image URI@hash to these blobs.

The blobs are built into a SIF file, and that is named by a hash over the manifest of the source container. We have to go out to get the manifest at present to calculate the hash and find if a matching image is cached.

We should ideally look at the cache first for OCI/Docker images specified with a URL@hash format - but we will have some work to get there that is beyond just implementing a simple change in flow with a flag.

[rhpvorderman]

Thank you for your answer. I understand this is hard and I also understand that this feature may never be implemented because of this. The docker registry was never designed to interoperate with singularity in the first place.

For now I have hacked together a workaround which can be used by other users as well who wish to work around this issue.

[carterpeel]

Hello,

This is a templated response that is being sent out to all open issues. We are working hard on 'rebuilding' the Singularity community, and a major task on the agenda is finding out what issues are still outstanding.

Please consider the following:

1. Is this issue a duplicate, or has it been fixed/implemented since being added?
2. Is the issue still relevant to the current state of Singularity's functionality?
3. Would you like to continue discussing this issue or feature request?

Thanks, Carter

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in over 60 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because no response was provided within 7 days.

[mr-c]

I can confirm that this is still an issue

$ singularity exec --containall docker://quay.io/biocontainers/fastqc@sha256:319b8d4eca0fc0367d192941f221f7fcd29a6b96996c63cbf8931dbb66e53348 fastqc --version
INFO:    Converting OCI blobs to SIF format
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
Getting image source signatures
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob 77c6c00e8b61 done  
Copying blob 3aaade50789a done  
Copying blob 00cf8b9f3d2a done  
Copying blob 7ff999a2256f done  
Copying blob d2ba336f2e44 done  
Copying blob dfda3e01f2b6 done  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob 10c3bb32200b done  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob 6d92b3a49ebf done  
Copying config 8b79910fc5 done  
Writing manifest to image destination
Storing signatures
2021/07/22 15:55:23  info unpack layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2021/07/22 15:55:23  info unpack layer: sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500
2021/07/22 15:55:23  warn rootless{dev/console} creating empty file in place of device 5:1
2021/07/22 15:55:23  warn xattr{home/default} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2021/07/22 15:55:23  warn xattr{/tmp/build-temp-802777424/rootfs/home/default} destination filesystem does not support xattrs, further warnings will be suppressed
2021/07/22 15:55:23  info unpack layer: sha256:3aaade50789a6510c60e536f5e75fe8b8fc84801620e575cb0435e2654ffd7f6
2021/07/22 15:55:23  info unpack layer: sha256:00cf8b9f3d2a08745635830064530c931d16f549d031013a9b7c6535e7107b88
2021/07/22 15:55:23  info unpack layer: sha256:7ff999a2256f84141f17d07d26539acea8a4d9c149fefbbcc9a8b4d15ea32de7
2021/07/22 15:55:23  info unpack layer: sha256:d2ba336f2e4458a9223203bf17cc88d77e3006d9cbf4f0b24a1618d0a5b82053
2021/07/22 15:55:23  info unpack layer: sha256:dfda3e01f2b637b7b89adb401f2f763d592fcedd2937240e2eb3286fabce55f0
2021/07/22 15:55:23  info unpack layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2021/07/22 15:55:23  info unpack layer: sha256:10c3bb32200bdb5006b484c59b5f0c71b4dbab611d33fca816cd44f9f5ce9e3c
2021/07/22 15:55:23  info unpack layer: sha256:6d92b3a49ebfad5fe895550c2cb24b6370d61783aa4f979702a94892cbd19077
INFO:    Creating SIF file...
WARNING: Skipping mount /home/michael/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
FastQC v0.11.9
# turn off networking and repeat
$ singularity exec --containall docker://quay.io/biocontainers/fastqc@sha256:319b8d4eca0fc0367d192941f221f7fcd29a6b96996c63cbf8931dbb66e53348 fastqc --version
FATAL:   Unable to handle docker://quay.io/biocontainers/fastqc@sha256:319b8d4eca0fc0367d192941f221f7fcd29a6b96996c63cbf8931dbb66e53348 uri: failed to get checksum for docker://quay.io/biocontainers/fastqc@sha256:319b8d4eca0fc0367d192941f221f7fcd29a6b96996c63cbf8931dbb66e53348: error pinging docker registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io: no such host
$ singularity version
3.8.0

[kmuriki]

Pending issues from the old repo are copied to the new repo (https://github.com/apptainer/apptainer/issues/1388) and cleaned from the old retired repo.