Script to fetch unique conda environments

[ewels]

Script to go over all conda environment.yml files and pull out a unique, normalised set (consistent whitespace, no comments).

Will be useful when requesting initial set of Seqera Containers for all modules.

669 * 4 = 2,676 Wave builds we need to request 😅 (Docker, Singularity, arch, arm).

Not really intending to merge this in as we hopefully won't need it long term, just putting it here for reference / so I can find it again.

[ewels]

I started putting together a script to do a first pull + bulk update of all modules. It deliberately runs slowly, waiting for container builds, so as not to hit rate limits.

I started running but remembered that when we introduce Conda lock files we will need new builds for all of these, no point in doing it yet.

Needs a little more work to extend to singularity / arm but should be about ready once that's done.

[famosab]

@ewels Is this still WIP? :)

[ewels]

Yup, afraid so - it's waiting for the other pieces of the puzzle to come together really. Getting closer now though!

[ewels]

I think that we can close this, I'm not sure that we'll ever actually want to run these scripts.

[ewels]

@mirpedrol here's the PR you can use. Suggested changes:

• [ ] Make it a CLI, take an argument for a single module so that you can run one, or several at a time
• [ ] Add --freeze to the Wave command so that images go to the Seqera Containers community registry
• [ ] Add logic to get https URLs
• [ ] Add logic to get conda lock files

[ewels]

See also this script - this has logic to get the Singularity https container URL. It might be possible to combine the two scripts even.