Extended container support

[bernt-matthias]

trafficstars

• [x] use yaml based job config (simplifies future extensions)
• [x] allow singularity in addition to docker
  • → tool developers do not need to install docker
• [x] allow to specify container resolver config file OR allow to enable/disable/configure certain container resolvers

@mvdbeek what would be your ideas on the 3rd point? Just a single option allowing to set a custom config file / multiple options? Would it be a good idea to use cached containers from CVMFS in IWC workflows .. or is there no advantage with this?

[mvdbeek]

• weekly CI can run with custom config that does not use the building solver (so we will recognize more easily if a container was not added to biocontainers)

there's a linter for that, no need ro run through tool tests.

• IWC could use singularity containers on CVMFS

I would not at all like to do that. Less cvmfs is good.

• → tool developers do not need to install docker

singularity is hard to install on linux and impossible to install on OSX

[bernt-matthias]

there's a linter for that, no need ro run through tool tests.

for IUC tool tests? Where is this defined? I never have seen this in action .. maybe its not working (https://github.com/galaxyproject/tools-iuc/issues/5240).

singularity is hard to install on linux and impossible to install on OSX

Hah. My impression on Ubuntu was exactly the opposite. singularity worked out of the box while docker needed lots of google searches and manually editing config files. But maybe I remember wrong.

Also thought that it might be good to use (or at least support) singularity in too tests because it seems to be the default choice for many installations .. basically most that run on a HPC, or? I guess OSX does not play a role for production systems (but only for some developers).

[mvdbeek]

Where is this defined?

planemo lint --biocontainers

[bernt-matthias]

planemo lint --biocontainers

So we should add this to the weekly CI? Obviously we can't ad it to the PR workflow.

Edit. But I guess planemo lint --biocontainers would trigger all the other linting which we decided that we don't it in weekly CI.

[mvdbeek]

I think that's a good idea, yeah.

[bernt-matthias]

Seems to work with planemo lint --biocontainers tools/circos -s tests,output,inputs,help,general,command,citations,tool_xsd .. thanks for the tip. I tried to implement this here: https://github.com/galaxyproject/tools-iuc/pull/5242/commits/7bfc486b11530bb4bb36ecf9148d14e6ba391aba

[bgruening]

We have new users that have local SIF files and would like to use Galaxy. It would be nice if we could have a good integration into planemo for that.

Singularity can be installed with conda on Linux.

[mvdbeek]

You still gotta set up one of the 3 things here: https://rootlesscontaine.rs/getting-started/apptainer/ . user namespaces are not that widely available on HPCs, the second option isn't really any safer than docker.

I said this for tool developers, for which installing docker is so much easier than installing singularity. I don't mind adding more support in planemo, but it's not an option for IWC or IUC.

[bernt-matthias]

Hi @bgruening I was able to run

planemo test --biocontainers --singularity --container_resolvers_config_file ./container_resolvers_dest.yml --no_dependency_resolution --no_conda_auto_init --galaxy_python_version 3.7 --no_cleanup TOOL_FILE

With ./container_resolvers_dest.yml:

- type: cached_explicit_singularity
  cache_directory: /tmp/container_cache/
- type: cached_mulled_singularity
  cache_directory: /tmp/container_cache/

Not sure yet if this enforces to use the containers in the cache or allows to fall back to conda.

Maybe your contacts could test this?

Image naming is a bit tricky. For instance for mulled containers the : must not be replaced by _ (which singularity pull does) and also the .sif extension should be removed.

[bernt-matthias]

Besides this we should have a few tests here. Could need a bit of inspiration where to put them etc...