[ndd] Convert docker image to singularity image

[tathey1]

from @jovo (full disclosure not currently aware of what this means)

[bvarjavand]

Docker and singularity are both container runtimes, which just means that they package the code you give them with its own computer. When you run code via docker or singularity, it starts up the small computer and then runs the code on it, which means your code will generate the same results anywhere.

We currently do not have a docker image of brainlit as there is no Dockerfile in the repo, but this could be accomplished with some development.
We want a singularity image as well since the JHU computing cluster MARCC is able to run singularity images.

[jovo]

right! we do have this: https://hub.docker.com/r/neurodata/cloudreg we will want a docker image that supersedes this one, to include all of brainlit and we'll want that docker to be trivially convertible into a singularity container, to enable it to run in both environments. or maybe only singularity, i just have less experience with singularity since i've only used docker myself.

On Fri, Jul 31, 2020 at 3:59 PM Bijan Varjavand [email protected] wrote:

•  External Email - Use Caution      *
  

Docker and singularity are both container runtimes, which just means that they package the code you give them with its own computer. When you run code via docker or singularity, it starts up the small computer and then runs the code on it, which means your code will generate the same results anywhere.

We currently do not have a docker image of brainlit as there is no Dockerfile in the repo, but this could be accomplished with some development. We want a singularity image as well since the JHU computing cluster MARCC is able to run singularity images.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fneurodata%2Fbrainlit%2Fissues%2F58%23issuecomment-667325353&data=02%7C01%7Cjovo%40jhu.edu%7Cd88be619b683432ab66808d8358c44c0%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637318223843611063&sdata=xhOg7ZzD674zQl1BXEoD%2FRTvaKncLd%2FoBafVS6J2V0c%3D&reserved=0, or unsubscribe https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAAKG4QUMYLB4BEDR7OE5ALR6MPC5ANCNFSM4PL2ZYVQ&data=02%7C01%7Cjovo%40jhu.edu%7Cd88be619b683432ab66808d8358c44c0%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637318223843611063&sdata=Wyc6AKmjwOsuwCjqC79iC%2BPSuki9CghgRQ1TDhYHI0c%3D&reserved=0 .

-- With gratitude,

Joshua T Vogelstein, PhD neurodata.io | BME@JHU https://www.bme.jhu.edu/ | dA/dt > 0 > dJ/dt https://twitter.com/neuro_data/status/1279067902658916352 where A = appreciating, J = judging, and t = time

[bvarjavand]

Indeed,

• You can shell into a Singularity-ized Docker image
• You can run a Docker image instantly as a Singularity image
• You can pull a Docker image (without sudo) So step 1 is to get the docker image.

[bvarjavand]

@jovo I've built an image with the working brainlit release here. Now to run via singularity

[bvarjavand]

Ok, the image is released at neurodata/brainlit. The code to shell, run, and execute the image via Singularity is

singularity shell docker://neurodata/brainlit:latest
singularity run docker://neurodata/brainlit:latest
singularity exec docker://neurodata/brainlit:latest echo "Brains!"

Once MARCC is back up I can test if it functions properly.

[jovo]

cool! it would be great if you could also help vikram with his docker, and/on simply include his stuff into yours...

On Mon, Aug 3, 2020 at 2:40 PM Bijan Varjavand [email protected] wrote:

•  External Email - Use Caution      *
  

Ok, the image is released at neurodata/brainlit. The code to run the image via Singularity is

singularity shell docker://neurodata/brainlit:latest singularity run docker://neurodata/brainlit:latest singularity exec docker://neurodata/brainlit:latest echo "Brains!"

Once MARCC is back up I can test if it functions properly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fneurodata%2Fbrainlit%2Fissues%2F58%23issuecomment-668179839&data=02%7C01%7Cjovo%40jhu.edu%7C4a101b4461674459a7b208d837dcab05%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637320768183412588&sdata=To%2BcLQg0UCqxgjA0HF1M%2BhZO%2FfjE1lvSUvYSxkgCyTY%3D&reserved=0, or unsubscribe https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAAKG4RGKYELAQUE7WJG4WLR64AA5ANCNFSM4PL2ZYVQ&data=02%7C01%7Cjovo%40jhu.edu%7C4a101b4461674459a7b208d837dcab05%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637320768183422583&sdata=F39EM5s4P18RxtMWjDXJKd2oXZE8CWFYI2uRpOK%2BByk%3D&reserved=0 .

-- With gratitude,

Joshua T Vogelstein, PhD neurodata.io | BME@JHU https://www.bme.jhu.edu/ | dA/dt > 0 > dJ/dt https://twitter.com/neuro_data/status/1279067902658916352 where A = appreciating, J = judging, and t = time

[bvarjavand]

Incorporating Vikram's stuff into brainlit is definitely the plan. Will do.

[JacopoTeneggi]

Hi! I tried running singularity shell docker://neurodata/brainlit:latestDocker image path: index.docker.io/neurodata/brainlit:latest on MARCC and the container is initialized correctly:

Creating container runtime...
Exploding layer: sha256:6ec8c9369e08152361a01729f2c8a1e7aae898426c6e67267f41894bf9524827.tar.gz
Exploding layer: sha256:401b5acb42e6784f873065f40787ef99fe6e18032e4a5a752699dde7c85af4c1.tar.gz
Exploding layer: sha256:0976b24edb456cdd2d97653c3ffa5d79b413457f128d30704c587ccb7a581504.tar.gz
Exploding layer: sha256:a338e6aafbddbf1c33e0126c272ec6661358feab53dc88c5d9aaa734f44db756.tar.gz
Exploding layer: sha256:b35ca89f21b64cd3b0764366449c3f42ada8f562c49b3041507088a1d57607de.tar.gz
Exploding layer: sha256:63f6f116cdd088bbabf4dc03213278c889e94f8a1dcb44c4d0e0994804fea8d1.tar.gz
Exploding layer: sha256:57c11c38cd22f1ada1f0214d49ccb659408feb03b5f6ce217e298ccf63847b65.tar.gz
Exploding layer: sha256:20ac95b9cde4e6cb4dd4d65fefa1a5b16a51d542755b537c1de70fa84fcceebc.tar.gz
Exploding layer: sha256:ed1f0677901e39f61e129c9a663f8bfcf3b60c5fd538914119581577737bcd53.tar.gz
Exploding layer: sha256:779214d62416fc8aaa1d4ffc2c538c2aeb9709da79d4db60fa60b7f723802853.tar.gz
Singularity: Invoking an interactive shell within container...

But when I try to import brainlit I get:

Singularity brainlit:latest:~> python
Python 3.8.5 (default, Jul 22 2020, 12:39:25) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import brainlit
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/brainlit/__init__.py", line 4, in <module>
    import brainlit.preprocessing
  File "/usr/local/lib/python3.8/site-packages/brainlit/preprocessing/__init__.py", line 1, in <module>
    from brainlit.preprocessing.image_process import *
  File "/usr/local/lib/python3.8/site-packages/brainlit/preprocessing/image_process.py", line 2, in <module>
    from skimage.measure import label
ModuleNotFoundError: No module named 'skimage'

I am going to build the Docker image locally to see if I can reproduce the error and try and debug it.