singularity

singularity copied to clipboard

Version of Singularity:

3.0.1-dirty (from conda forge)

Expected behavior

Without internet connection: singularity pull docker://ubuntu hangs indefinitely singularity run docker://ubuntu hangs indefinitely

Actual behavior

Many HPC worker nodes don't have internet access. However, to use singularity as a drop-in replacement for docker, it should have some kind of fallback for when it doesn't have internet.

When you singularity run docker://ubuntu, I think the best behaviour is to:

• Try to pull the image. If this succeeds then run it immediately
• If this fails (due to timeout), check the cache, and if any version of ubuntu that matches the image spec is there, run it (ie this would fail if I said singularity run docker://ubuntu:18.04 and my cache only had an Ubuntu 16.04 image).
• If the image is not in the cache, print out an error message and exit with nonzero status

When you singularity pull docker://ubuntu, I think the best behaviour is to:

• Try to pull the image. If this succeeds then exit 0
• If this times out and an appropriate image is in the cache, print a message explaining this, and exit 0 (so that the user/any controlling script knows that there's still the option to singularity run this image)
• If the image is not in the cache, print out an error message and exit with nonzero status

Steps to reproduce behavior

Run: singularity pull docker://ubuntu without internet

I can reproduce this with:

env HTTPS_PROXY=http://127.0.0.1:8000/ SINGULARITY_CACHEDIR=$PWD/tmp/cache builddir/singularity -d pull --force docker://alpine:3.9

There's no proxy on that port, so it's acting as if the network was doing (it's a different code path in the http package, so it's not the exact same thing).

But I could easily make something returning 404.

Ignoring an implementation detail for a second, I think on that last point lies the problem: if you get a 404 from the server, do you still go to the cache?

I believe without a connection, most proxies will return 503. Do I go to the cache in that case?

@TMiguelT, my general impression is that your use case requires a caching proxy, not singularity falling back to the cache.

The implementation detail I'm ignoring is the the way the singularity cache works right now, which is a a little underspecified. It's mostly there to save bandwidth: it queries the remote server, asks for a hash, compares against the local version, and if it's the same hash, it skips the full download. When the remote server is not available, it cannot obtain the hash and therefore it does not work. The integrity of the contents is handled using cryptographic signatures.

I'll make a note to test something with a proxy.

Hello,

This is a templated response that is being sent out to all open issues. We are working hard on 'rebuilding' the Singularity community, and a major task on the agenda is finding out what issues are still outstanding.

Please consider the following:

1. Is this issue a duplicate, or has it been fixed/implemented since being added?
2. Is the issue still relevant to the current state of Singularity's functionality?
3. Would you like to continue discussing this issue or feature request?

Thanks, Carter

This issue has been automatically marked as stale because it has not had activity in over 60 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@multimeric Still getting this problem ? Have this been solved already ? If yes, what work around have you followed and applied ?

We're looking into the issue carefully, soon will bring to community and discuss ways to better solve as well address this. Thankyou for keeping the interest in the subject.

Old Singularity repo issues are now closed. If this is still critical please open a new issue in Apptainer repo.