modules icon indicating copy to clipboard operation
modules copied to clipboard

[FEATURE] Update modules to remove undocumented task.ext fields

Open mahesh-panchal opened this issue 6 months ago • 1 comments

Is your feature request related to a problem? Please describe

task.ext fields were introduced allow users to override developer settings in pipelines. However some modules are simply using them as an alternative input: method for parameters that don't always need to be supplied.

% find modules -name "*.nf" -exec grep -H "task.ext." {} \; | grep -vE "task\.ext\.(singularity|args|prefix|when|suffix)"
modules/nf-core/picard/fixmateinformation/main.nf:    def STRINGENCY = task.ext.stringency ?: "STRICT"
modules/nf-core/artic/minion/main.nf:    def hd5_plugin_path = task.ext.hd5_plugin_path ? "export HDF5_PLUGIN_PATH=${task.ext.hd5_plugin_path}" : "export HDF5_PLUGIN_PATH=/usr/local/lib/python3.6/site-packages/ont_fast5_api/vbz_plugin"
modules/nf-core/fgbio/zipperbams/main.nf:    def compression = task.ext.compression ?: '0'
modules/nf-core/jupyternotebook/main.nf:    def parametrize = (task.ext.parametrize == null) ?  true : task.ext.parametrize
modules/nf-core/jupyternotebook/main.nf:    def implicit_params = (task.ext.implicit_params == null) ? true : task.ext.implicit_params
modules/nf-core/jupyternotebook/main.nf:    def meta_params = (task.ext.meta_params == null) ? true : task.ext.meta_params
modules/nf-core/jupyternotebook/main.nf:    def kernel   = task.ext.kernel ?: '-'
modules/nf-core/rmarkdownnotebook/main.nf:    def parametrize = (task.ext.parametrize == null) ?  true : task.ext.parametrize
modules/nf-core/rmarkdownnotebook/main.nf:    def implicit_params = (task.ext.implicit_params == null) ? true : task.ext.implicit_params
modules/nf-core/rmarkdownnotebook/main.nf:    def meta_params = (task.ext.meta_params == null) ? true : task.ext.meta_params
modules/nf-core/plink2/vcf2bgen/main.nf:    def reffirst = bgen_reffirst || task.ext.reffirst ? "ref-first" : ""
modules/nf-core/plink2/vcf2bgen/main.nf:    def dosage = task.ext.dosage_field ? task.ext.dosage_field : dosage_field
modules/nf-core/plink2/vcf2bgen/main.nf:    def sample_name_opt = task.ext.sample_name_mode ? task.ext.sample_name_mode : sample_name_mode
modules/nf-core/hmmer/eslreformat/main.nf:    def postproc = task.ext.postprocessing ?: ""
modules/nf-core/cellbender/merge/main.nf:    output_layer = task.ext.output_layer ?: "cellbender"
modules/nf-core/cellbender/removebackground/main.nf:    container "${ task.ext.use_gpu ? 'us.gcr.io/broad-dsde-methods/cellbender:0.3.2' :
modules/nf-core/cellbender/removebackground/main.nf:    use_gpu = task.ext.use_gpu ? "--cuda" : ""
modules/nf-core/bedtools/groupby/main.nf:    def summary_col = task.ext.summary_col ? "-c ${task.ext.summary_col}" : "-c 5"
modules/nf-core/scvitools/scar/main.nf:    input_layer = task.ext.input_layer ?: "X"
modules/nf-core/scvitools/scar/main.nf:    output_layer = task.ext.output_layer ?: "scar"
modules/nf-core/scvitools/scar/main.nf:    max_epochs = task.ext.max_epochs ?: ""
modules/nf-core/scvitools/solo/main.nf:    container "${ task.ext.use_gpu ? 'docker.io/nicotru/scvitools-gpu:cuda-12' :
modules/nf-core/scvitools/solo/main.nf:    batch_key = task.ext.batch_key ?: ""
modules/nf-core/scvitools/solo/main.nf:    max_epochs = task.ext.max_epochs ?: ""
modules/nf-core/metaspace/converter/main.nf:    database_name    = task.ext.database_name ?: "HMDB"
modules/nf-core/metaspace/converter/main.nf:    database_version = task.ext.database_version ?: "v4"
modules/nf-core/metaspace/converter/main.nf:    fdr              = task.ext.fdr ?: "0.1"
modules/nf-core/metaspace/converter/main.nf:    use_tic          = task.ext.use_tic ?: "true"
modules/nf-core/metaspace/converter/main.nf:    metadata_as_obs  = task.ext.metadata_as_obs ?: "false"
modules/nf-core/sentieon/dedup/main.nf:    def metrics = task.ext.metrics ?: "${prefix}.metrics"
modules/nf-core/sentieon/dedup/main.nf:    def metrics = task.ext.metrics ?: "${prefix}.metrics"
modules/nf-core/icountmini/segment/main.nf:    def regions_prefix = task.ext.regions_prefix ?: "${gtf.simpleName}"
modules/nf-core/icountmini/segment/main.nf:    def regions_prefix = task.ext.regions_prefix ?: "${gtf.simpleName}"

Describe the solution you'd like

The modules should be refactored so the "non-standard" task.ext fields are moved to input: and documented in the meta.yml so pipeline developers know what to set.

Describe alternatives you've considered

No response

Additional context

No response

mahesh-panchal avatar Jun 25 '25 14:06 mahesh-panchal

task.ext.use_gpu was added in https://github.com/nf-core/tools/pull/3272

From Nico: https://nfcore.slack.com/archives/C043UU89KKQ/p1750860014339069?thread_ts=1750840167.300249&cid=C043UU89KKQ

The basic idea is that process_gpu signalises that a process supports running on a GPU. The task.ext.use_gpu is then used to signalize to the process that it actually should use it. This will generally be the case if the gpu profile is enabled. Thus, the GPU profile is "forwarded" via task.ext.use_gpu. However, in some cases one might want to use the GPU in general, but for some processes the GPU does not have enough VRAM or something. Then the user can override task.ext.use_gpu for single processes.

mahesh-panchal avatar Jun 25 '25 14:06 mahesh-panchal