RajLabMSSM
Singularity image fail to start
1. Bug description
Our cluster does not support Docker, so we would like to use the Singularity image.
Unfortunately, it fail to start with the following error message:
s6-overlay-preinit: fatal: unable to chown /var/run/s6: Read-only file system
Expected behaviour
The image should run.
2. Reproducible example
Code
$ singularity pull docker://bschilder/echolocator
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob b50f1017556e done
Copying blob ac2e64a9e36a done
Copying blob e1b2b3a6a723 done
Copying blob 3e1633226ea3 done
Copying blob b3cacfb902ee done
Copying blob e96e057aae67 done
Copying blob ee6ca148ef07 done
Copying blob 1b92e41706af done
Copying blob 4c84da83eb41 done
Copying blob ad8ee9a4fa31 done
Copying blob 64c8c817d8b1 done
Copying blob ba3f6b7e0f33 done
Copying blob ab4919bad7d1 done
Copying blob 1bd3d7f33b43 done
Copying blob 8aa30d70216c done
Copying config bb2d532c8c done
Writing manifest to image destination
Storing signatures
2023/01/09 12:33:47 info unpack layer: sha256:e96e057aae67380a4ddb16c337c5c3669d97fdff69ec537f02aa2cc30d814281
2023/01/09 12:33:48 info unpack layer: sha256:3e1633226ea37a1c4c5f94733ed7d77c26f5d5714300ea8363dd096916184c75
2023/01/09 12:33:48 info unpack layer: sha256:e1b2b3a6a7232a3aae8f678c48c3188e7fb87a8ba5942ee82cb3f233659d6d86
2023/01/09 12:33:58 info unpack layer: sha256:ac2e64a9e36af4cdbc00c3639016e2382063af7738af64333b3a85cf77d02048
2023/01/09 12:33:59 info unpack layer: sha256:b3cacfb902eef0b6c03796b9e6e68711244dadcb14191371e4c2debc83d22240
2023/01/09 12:34:00 info unpack layer: sha256:b50f1017556e64d2d42f9c965fbce3e4067463e14029034358041449ed2f9899
2023/01/09 12:34:13 info unpack layer: sha256:ee6ca148ef071114d6c4999a2b7021b6a7e2a25b177192fe2dc05f18ed6fccc7
2023/01/09 12:34:13 info unpack layer: sha256:1b92e41706af18902da960e0eb198337c600a94b87d0660786588db3d9009b5b
2023/01/09 12:34:13 info unpack layer: sha256:4c84da83eb41120cd68c4fb81638fafe032723f8798d1e12d3d522ca9a48fb57
2023/01/09 12:34:13 info unpack layer: sha256:ad8ee9a4fa311d989b0701ef5553d9b0a63dd789a2100045cead08a45527549e
2023/01/09 12:34:56 info unpack layer: sha256:64c8c817d8b13243e15fa9b4af37f481f81c3d0f55679ccb493a18b96b67ddeb
2023/01/09 12:34:56 info unpack layer: sha256:ba3f6b7e0f33bbd418530ec08c450e84a8fc71b155110951bd747a1c9c50429c
2023/01/09 12:34:56 info unpack layer: sha256:ab4919bad7d196e713faf86f24f78f9ebf7170633ba8c19f367297a81f01cada
2023/01/09 12:34:59 info unpack layer: sha256:1bd3d7f33b437c11b03aeaa7d7edbdc0cc28b90f630c2e69d1598decc2e73a60
2023/01/09 12:34:59 info unpack layer: sha256:8aa30d70216cc13500ca458121b3c59aa08a2315629a78ce20c088869652823d
2023/01/09 12:35:44 warn rootless{usr/local/lib/R/site-library/Rhtslib/usrlib/libhts.so.2} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
INFO: Creating SIF file...
$ ./echolocator_latest.sif
s6-overlay-preinit: fatal: unable to chown /var/run/s6: Read-only file system
Environment
- OS: Debian GNU/Linux 10 (buster)
- Singularity version: 3.10.4
Console output
see above
Data
not applicable
3. Session info
not applicable
hi @mfasco , this sounds like a permissions issue with how your HPC has set up singularity. i think the best course would be to reach out to your HPC support team directly.
hi @bschilder, thanks for the quick reply. I actually replicated the same issue on my desktop, which has Ubuntu 18.04 and the latest version of Singularity, 3.10.4.
Thanks.
sorry to bother you again @bschilder - I would like to help debug the problem - is there a way to access the Dockerfile and all it's needed to build the image?
hi @bschilder, thanks for the quick reply. I actually replicated the same issue on my desktop, which has Ubuntu 18.04 and the latest version of Singularity, 3.10.4.
Thanks for this info @mfasco, this helps narrow down the possibilities.
Sorry haven't had a chance to look into this further just yet. WIll try to get to this soon.
The Docker image that is currently on Dockerhub is one that I created manually (without a Dockerfile) by pulling the Bioc docker image: https://www.bioconductor.org/help/docker/
I'm working on automating this with my rworkflows GitHub Action, which uses a generic Dockerfile. You can access a copy of this Dockerfile with rworkflows::use_dockerfile() from the rworkflows R package, which can also be installed from here.
I've just launched a new GitHub Actions run for echolocatoR, so if all goes well, hopefully the new image will be be up soon on Dockerhub. I'll keep you posted!
Mac
I've just tried regenerating the singularity container on my local Mac:
> singularity pull docker://bschilder/echolocator
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob sha256:e96e057aae67380a4ddb16c337c5c3669d97fdff69ec537f02aa2cc30d814281
29.02 MiB / 29.02 MiB [====================================================] 1s
Copying blob sha256:3e1633226ea37a1c4c5f94733ed7d77c26f5d5714300ea8363dd096916184c75
1.68 KiB / 1.68 KiB [======================================================] 0s
Copying blob sha256:e1b2b3a6a7232a3aae8f678c48c3188e7fb87a8ba5942ee82cb3f233659d6d86
259.41 MiB / 259.41 MiB [=================================================] 13s
Copying blob sha256:ac2e64a9e36af4cdbc00c3639016e2382063af7738af64333b3a85cf77d02048
19.29 KiB / 19.29 KiB [====================================================] 0s
Copying blob sha256:b3cacfb902eef0b6c03796b9e6e68711244dadcb14191371e4c2debc83d22240
26.29 MiB / 26.29 MiB [====================================================] 0s
Copying blob sha256:b50f1017556e64d2d42f9c965fbce3e4067463e14029034358041449ed2f9899
235.53 MiB / 235.53 MiB [=================================================] 10s
Copying blob sha256:ee6ca148ef071114d6c4999a2b7021b6a7e2a25b177192fe2dc05f18ed6fccc7
26.60 KiB / 26.60 KiB [====================================================] 0s
Copying blob sha256:1b92e41706af18902da960e0eb198337c600a94b87d0660786588db3d9009b5b
52.83 KiB / 52.83 KiB [====================================================] 0s
Copying blob sha256:4c84da83eb41120cd68c4fb81638fafe032723f8798d1e12d3d522ca9a48fb57
1.18 KiB / 1.18 KiB [======================================================] 0s
Copying blob sha256:ad8ee9a4fa311d989b0701ef5553d9b0a63dd789a2100045cead08a45527549e
832.87 MiB / 832.87 MiB [=================================================] 45s
Copying blob sha256:64c8c817d8b13243e15fa9b4af37f481f81c3d0f55679ccb493a18b96b67ddeb
272 B / 272 B [============================================================] 0s
Copying blob sha256:ba3f6b7e0f33bbd418530ec08c450e84a8fc71b155110951bd747a1c9c50429c
255 B / 255 B [============================================================] 0s
Copying blob sha256:ab4919bad7d196e713faf86f24f78f9ebf7170633ba8c19f367297a81f01cada
73.90 MiB / 73.90 MiB [====================================================] 3s
Copying blob sha256:1bd3d7f33b437c11b03aeaa7d7edbdc0cc28b90f630c2e69d1598decc2e73a60
1.44 KiB / 1.44 KiB [======================================================] 0s
Copying blob sha256:8aa30d70216cc13500ca458121b3c59aa08a2315629a78ce20c088869652823d
950.35 MiB / 950.35 MiB [=================================================] 28s
Copying config sha256:bb2d532c8cbc41b332b9bbe82ac6f9d5accb2bf2e202e7cf4ad5fb8f9a7829b3
10.24 KiB / 10.24 KiB [====================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: echolocator_latest.sif
When i try the command I get a different error, but this has to do with Singularity not being supported on Mac:
> ./echolocator_latest.sif
vmx_init: processor does not support desired secondary processor-based controls
The issue has something to do with how my VM is setup, which I'll also need to follow up on.
Linux
So I tried generating a new container on my HPC, which is Linux/CentOS-based:
> singularity pull docker://bschilder/echolocator
INFO: Detected Singularity user configuration directory
INFO: Detected public Singularity pgp keyring, migrating...
INFO: Detected private Singularity pgp keyring, migrating...
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob b50f1017556e done
Copying blob e96e057aae67 done
Copying blob e1b2b3a6a723 done
Copying blob b3cacfb902ee done
Copying blob ac2e64a9e36a done
Copying blob 3e1633226ea3 done
Copying blob ee6ca148ef07 done
Copying blob 1b92e41706af done
Copying blob 4c84da83eb41 done
Copying blob ad8ee9a4fa31 done
Copying blob 64c8c817d8b1 done
Copying blob ba3f6b7e0f33 done
Copying blob ab4919bad7d1 done
Copying blob 1bd3d7f33b43 done
Copying blob 8aa30d70216c done
Copying config bb2d532c8c done
Writing manifest to image destination
Storing signatures
2023/01/20 14:45:11 info unpack layer: sha256:e96e057aae67380a4ddb16c337c5c3669d97fdff69ec537f02aa2cc30d814281
2023/01/20 14:45:49 info unpack layer: sha256:3e1633226ea37a1c4c5f94733ed7d77c26f5d5714300ea8363dd096916184c75
2023/01/20 14:45:49 info unpack layer: sha256:e1b2b3a6a7232a3aae8f678c48c3188e7fb87a8ba5942ee82cb3f233659d6d86
2023/01/20 14:47:45 info unpack layer: sha256:ac2e64a9e36af4cdbc00c3639016e2382063af7738af64333b3a85cf77d02048
2023/01/20 14:47:46 info unpack layer: sha256:b3cacfb902eef0b6c03796b9e6e68711244dadcb14191371e4c2debc83d22240
2023/01/20 14:47:49 info unpack layer: sha256:b50f1017556e64d2d42f9c965fbce3e4067463e14029034358041449ed2f9899
2023/01/20 14:49:56 info unpack layer: sha256:ee6ca148ef071114d6c4999a2b7021b6a7e2a25b177192fe2dc05f18ed6fccc7
2023/01/20 14:49:57 info unpack layer: sha256:1b92e41706af18902da960e0eb198337c600a94b87d0660786588db3d9009b5b
2023/01/20 14:49:57 info unpack layer: sha256:4c84da83eb41120cd68c4fb81638fafe032723f8798d1e12d3d522ca9a48fb57
2023/01/20 14:49:57 info unpack layer: sha256:ad8ee9a4fa311d989b0701ef5553d9b0a63dd789a2100045cead08a45527549e
2023/01/20 14:58:19 info unpack layer: sha256:64c8c817d8b13243e15fa9b4af37f481f81c3d0f55679ccb493a18b96b67ddeb
2023/01/20 14:58:19 info unpack layer: sha256:ba3f6b7e0f33bbd418530ec08c450e84a8fc71b155110951bd747a1c9c50429c
2023/01/20 14:58:19 info unpack layer: sha256:ab4919bad7d196e713faf86f24f78f9ebf7170633ba8c19f367297a81f01cada
2023/01/20 14:59:21 info unpack layer: sha256:1bd3d7f33b437c11b03aeaa7d7edbdc0cc28b90f630c2e69d1598decc2e73a60
2023/01/20 14:59:21 info unpack layer: sha256:8aa30d70216cc13500ca458121b3c59aa08a2315629a78ce20c088869652823d
2023/01/20 15:03:30 warn rootless{usr/local/lib/R/site-library/Rhtslib/usrlib/libhts.so.2} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
INFO: Creating SIF file...
Then ran this command to access the terminal within the container:
./echolocator_latest.sif exec bash
After an initial warning (seems unrelated to your issue), I'm able to access the terminal within the container and open R:
couldn't read file "/apps/modules/4.7.1/libexec/modulecmd.tcl": no such file or directory
Apptainer> R
WARNING: ignoring environment value of R_HOME
R version 3.6.1 (2019-07-05) -- "Action of the Toes"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
That said, it doesn't seem to have echolocatoR within it! Or at least it can't find the R lib to call it:
library(echolocatoR)
# Error in library(echolocatoR) : there is no package called ‘echolocatoR’
This may be resolved by using the Dockerfile-bases approach that rworkflows uses. Last run failed due to the Dockerfile file not getting passed the PAT_GITHUB token (and some rtracklayer dependency issues) but I'll try to fix this and rerun:
https://github.com/RajLabMSSM/echolocatoR/actions/runs/3968409612/jobs/6801540238