Support custom labels when building containers with Conda and Spack packges

[pditommaso]

A key capability of Wave is the build of container by providing one or more Conda or Spack packages.

When relying on this mechanism the user has no control on the labels annotation of the resulting container.

The goal of this feature is to allow the use of custom labels in the container build request.

[marcodelapierre]

Ideas from the Wave meeting:

• arbitrary labels
• eventually, also pre-canned default labels, e.g. list of included conda packages+versions

[marcodelapierre]

For discussion

First hypothesis for arbitrary labels a label option (note each label is a key-value pair):

• Wave Client in Nextflow: a wave.build.conda.labels that allows to specify a map
• Wave CLI: a --label option, that can be inserted multiple times to provide multiple pairs

We should also think about pre-defined labels, for instance, when a list of packages is specified we could have a packages label providing pkg name+version

• see for instance this old draft by Vanessa at the Spack team
• another example from BioContainers
• not immediate to translate into case of provided Conda/Spack env file
• needs a toggle to disable?

[pditommaso]

All fine, just wave.build.labels instead of wave.build.conda.labels

[munishchouhan]

PR in libseqera created https://github.com/seqeralabs/libseqera/pull/18

[munishchouhan]

three PR created

https://github.com/seqeralabs/libseqera/pull/18 https://github.com/seqeralabs/wave-cli/pull/59 https://github.com/seqeralabs/wave/pull/422 and tested in local

 % ./app/build/native/nativeCompile/wave -f Dockerfile --wave-endpoint https://52cfbbd2458d.ngrok.app --label key=value --label "key2"="value2"
52cfbbd2458d.ngrok.app/wt/60b51493e886/hrma017/dev:df3ae06b3bcfaac7
munish.chouhan@Munishs-MacBook-Pro wave-cli % docker pull 52cfbbd2458d.ngrok.app/wt/60b51493e886/hrma017/dev:df3ae06b3bcfaac7
df3ae06b3bcfaac7: Pulling from wt/60b51493e886/hrma017/dev
8a1e25ce7c4f: Pull complete 
8ab039a68e51: Pull complete 
2b12a49dcfb9: Pull complete 
cdf9868f47ac: Pull complete 
e73ea5d3136b: Pull complete 
890ad32c613f: Pull complete 
4f4fb700ef54: Pull complete 
ba517b76f92b: Pull complete 
Digest: sha256:1c2a765b68dc03ce81b86955d72218cc4673eccc16d41a4e40cbef7941df69b9
Status: Downloaded newer image for 52cfbbd2458d.ngrok.app/wt/60b51493e886/hrma017/dev:df3ae06b3bcfaac7
52cfbbd2458d.ngrok.app/wt/60b51493e886/hrma017/dev:df3ae06b3bcfaac7
munish.chouhan@Munishs-MacBook-Pro wave-cli % docker inspect 52cfbbd2458d.ngrok.app/wt/60b51493e886/hrma017/dev:df3ae06b3bcfaac7 --format '{{ index .Config.Labels }}'
map[key:value key2:value2]

[marcodelapierre]

Thank you Munish, I have left some comments inside the PRs.

[marcodelapierre]

@pditommaso I assume we want the functionality available through Nextflow as well?

If so, we need to also add support for a Wave plugin option like wave.build.label

[munishchouhan]

@pditommaso I assume we want the functionality available through Nextflow as well?

If so, we need to also add support for a Wave plugin option like wave.build.label

@marcodelapierre Yes, I started working on it too, But not 100% sure, where this property gets defined in wave plugin

[munishchouhan]

@pditommaso @marcodelapierre I have created a draft PR in naxtflow, but I am not sure, if its correct or what else needs to be done on it. please advise https://github.com/nextflow-io/nextflow/pull/4817

[marcodelapierre]

Thank you, I will comment there.

[edit] Actually, it is more relevant here.

Regarding the draft PR for Nexflow, I have just given a high level, and overall the strategy looks correct to me.

However, I have one main suggestion for the high level design of all these PRs.
You don't need to add a new labels attribute to SubmitContainerTokenRequest, instead you can leverage the existing containerConfig, which is already used for a number of parameters (as you can see from the class def in the Nextflow codebase): env, cmd, entrypoint, layers.

In your draft impl in Nextflow, you have leveraged this class to add a labels attribute, and I think it is a good idea. Now, consider taking this as a key design choice, and use the nextflowConfig object to carry around the Labels info between the various components (libseqera, cli, wave, ...). It looks like you can keep taking inspiration from how the layers functionality is implemented.

What do you think?

[munishchouhan]

@marcodelapierre thank you, I will make the changes

[munishchouhan]

Test after moving labels to conatiner config for conda image

munish.chouhan@Munishs-MacBook-Pro wave-cli % ./app/build/native/nativeCompile/wave  --conda-package='salmon' --wave-endpoint https://7c70f9f53bee.ngrok.app --label key=value --label "key2"="value2"
7c70f9f53bee.ngrok.app/wt/85551871463d/hrma017/dev:salmon--88bbb93b1e6558b4
munish.chouhan@Munishs-MacBook-Pro wave-cli % docker inspect 7c70f9f53bee.ngrok.app/wt/85551871463d/hrma017/dev:salmon--88bbb93b1e6558b4 --format '{{ index .Config.Labels }}'
map[key:value key2:value2 org.opencontainers.image.created:2023-12-22T06:51:19.576Z org.opencontainers.image.description:Rapid builds of small Conda-based containers using micromamba. org.opencontainers.image.licenses:Apache-2.0 org.opencontainers.image.revision:bbc3ef3862d3f4568d55706dbe6e8129f88f6325 org.opencontainers.image.source:https://github.com/mamba-org/micromamba-docker org.opencontainers.image.title:micromamba-docker org.opencontainers.image.url:https://github.com/mamba-org/micromamba-docker org.opencontainers.image.version:latest]

[munishchouhan]

Test with Singularity image of conda packages

munish.chouhan@Munishs-MacBook-Pro wave-cli % docker run  --rm -v /tmp:/opt  quay.io/singularity/singularity:v3.11.4-slim-arm64 inspect /opt/image.sif 
key: value
key2: value2
org.label-schema.build-arch: arm64
org.label-schema.build-date: Wednesday_13_March_2024_20:35:44_UTC
org.label-schema.schema-version: 1.0
org.label-schema.usage.singularity.deffile.bootstrap: docker
org.label-schema.usage.singularity.deffile.from: mambaorg/micromamba:1.5.5
org.label-schema.usage.singularity.version: 3.11.4
org.opencontainers.image.created: 2023-12-22T06:51:19.576Z
org.opencontainers.image.description: Rapid builds of small Conda-based containers using micromamba.
org.opencontainers.image.licenses: Apache-2.0
org.opencontainers.image.revision: bbc3ef3862d3f4568d55706dbe6e8129f88f6325
org.opencontainers.image.source: https://github.com/mamba-org/micromamba-docker
org.opencontainers.image.title: micromamba-docker
org.opencontainers.image.url: https://github.com/mamba-org/micromamba-docker
org.opencontainers.image.version: latest

[marcodelapierre]

Thank you for the updates Munish. I have started with Libseqera, will do Wave/Wave CLI next.

Note that Paolo is keen on prioritising the build id refactor, which edits some of the same source files impacted by this issue.