modules icon indicating copy to clipboard operation
modules copied to clipboard

build: Remove conda env name

Open edmundmiller opened this issue 1 year ago • 6 comments

Originally trying to reproduce the error in https://github.com/nf-core/modules/issues/4234#issue-1961683896

It did get produced.

Figured the easiest path is actually to just stop testing conda and pytest-workflow together. I think it's a worthy sacrifice to move this forward, since we're phasing pytest-workflow out soon(™️)

This PR removes a few env names, and changes the linting to not require the name field, and removes the defaults channel (Take it up with @pditommaso if you liked the defaults channel)

edmundmiller avatar Jun 12 '24 16:06 edmundmiller

https://github.com/nextflow-io/nextflow/issues/4464#issuecomment-2160348935 Might be relevant

edmundmiller avatar Jun 12 '24 17:06 edmundmiller

Removed conda pytest-workflow tests from running.

It creates a race condition with the environment creation that it appears nf-test doesn't run into.

Keeping it prevents us from removing the env name from the environment.yml files, which is preventing us from having consistent image naming with Seqera containers, which is preventing us from building our "mulled" containers with Wave and using the Seqera container registry.

edmundmiller avatar Jun 12 '24 22:06 edmundmiller

If I am following this correctly, we not only want name to not be required but we want to avoid having a name in invironment.yml, is this right? In such case we should add a listing test for this.

Also, would it be better to remove the name progressively with PRs where we start using wave?

Apart from that, I agree with not using conda with pytest.

so basically the opposite of the current linting?

maxulysse avatar Jun 13 '24 15:06 maxulysse

so basically the opposite of the current listing?

Yes! We can try to add a not in the JSON schema

"required": ["channels", "dependencies"],
"not": { "required": [ "name" ] }

mirpedrol avatar Jun 13 '24 16:06 mirpedrol

The main problem I see with this is that the error pytest-workflow is throwing isn't limited to just pytest-workflow. It's also a real-life bug that can hit end users. So by removing name we potentially open up conda users to hitting the same error.

The workaround is not too difficult: use mamba. But it's not ideal. But I can't think of any way around it, given that we need to remove name for the Wave generated container name consistency..

ewels avatar Jun 19 '24 06:06 ewels

So the person who's going to run into issues with this:

  1. Has multiple users/pipelines going on the same machine
  2. Is using Conda
  3. Can't use mamba(mambaforge should work for just about any user space)

The other option is we convince the Seqera containers people to change their default so we can keep our environment names.

The issue isn't really with Nextflow either, it's a conda problem.

I think the point I'm trying to make is anyone who has those three requirements is pretty niche. I want nf-core to try to support everyone, but that's a pretty inflexible system, and we would have provided that person with multiple solutions.

Another thought, what if we changed the environment names that Nextflow autogenerated to match the wave default? Like a spike-in at runtime. That should work in theory if it works when we give it an environment name. Could be a backlog item and we see if our stubborn conda users with multiple people running it come out of the woodwork...

edmundmiller avatar Jun 29 '24 17:06 edmundmiller

Done in https://github.com/nf-core/modules/pull/6600

WITH THE EXCEPTION OF REMOVING THE CONDA PROFILE.

So may need to add this change still.

ewels avatar Sep 06 '24 08:09 ewels

Note that the conda error should be fine with micromamba. Currently some issues with that (see https://github.com/nextflow-io/nextflow/issues/4161 ) but hoping to get fixed soon.

So maybe we can continue testing with conda without this problem by switching to mamba, if needed.

ewels avatar Sep 06 '24 09:09 ewels