Caching intermediate outputs in CWL workflows like `cwltool --cachedir`

[arunisaac]

Hi,

This is a feature request. I am running CWL workflows on HPC (slurm and grid engine) using toil. I would like for toil to cache outputs from intermediate steps and reuse them later in subsequent runs of the workflow. cwltool can do this already with its --cachedir argument. This feature is very helpful to avoid recomputing steps when incrementally developing and troubleshooting workflows. The closest toil has to this feature is to combine the --clean never and --restart flags. But, this is not robust to changes in the workflow. It only works for rerunning the exact same workflow, not slightly modified versions.

Thank you!

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1523