toil icon indicating copy to clipboard operation
toil copied to clipboard

CWL: config file for always-needed toil-cwl-runner options

Open mr-c opened this issue 2 years ago • 3 comments

Many sites that run toil-cwl-runner, especially HPC systems, need quite a lot of options defined. It would be nice if there was a config file that formed the basis of the command-line options.

See https://github.com/EBI-Metagenomics/emg-viral-pipeline/blob/f367002f0e1e375840e5696323bde65f7accb31f/cwl/virify.sh#L300 for an advanced script from a toil-cwl-runner user

Where? In toil-cwl. + ini or toml or somesuch; under ${XDG_CONFIG_HOME} (which defaults to ${HOME}/.config)

Additional command line options should have priority. The location of the config file should also be overridable with an environment variable.

┆Issue is synchronized with this Jira Story ┆friendlyId: TOIL-1186

mr-c avatar Jun 15 '22 16:06 mr-c

We have wanted a config file for a while; we should be able to hook into some of the code we have already to pull from either the command line or environment variables.

It might also be nice if it could integrate well with user Toil workflows, so e.g. Cactus or toil-vg could ride the same system.

Lon had the idea of a config generator to show you all the defaults.

@mr-c Do you want to implement this system?

adamnovak avatar Jun 30 '22 16:06 adamnovak

@stxue1 Did your new config file stuff in #4569 address this?

adamnovak avatar Oct 05 '23 17:10 adamnovak

It looks like the config file does not work for toil-cwl-runner. I think configargparse has an issue with nargs=argparse.REMAINDER. It looks like if another nargs/append action is in the mix, the REMAINDER argparse option swallows all config file options.

stxue1 avatar Oct 09 '23 18:10 stxue1