Toil can't override inputs section values from the WDL input JSON

[adamnovak]

In https://github.com/miramastoras/hpp_production_workflows/blob/ab8bb603ead7c6d38f9d3e21518e7871dd5772b6/QC/wdl/workflows/hprc_DeepPolisher.wdl we call the PHARAOH workflow, which calls a task with threadCount=64.

If you try and call it with an inputs JSON that sets hprc_DeepPolisher.PHARAOH.alignmentPaf.threadCount to override that hardcoded input value, Toil throws an error from inside MiniWDL saying that that's not a workflow input:

+ toil-wdl-runner --batchSystem single_machine --batchLogsDir ./toil_logs /private/groups/hprc/polishing/hpp_production_workflows/QC/wdl/workflows/hprc_DeepPolisher.wdl /data/tmp/anovak/HPRC_DeepPolisher_HG02015/HG02015_hprc_DeepPolisher.json --outputDirectory hprc_DeepPolisher_outputs --outputFile HG02015_hprc_DeepPolisher_outputs.json --runLocalJobsOnWorkers --retryCount 1 --disableProgress=True
Traceback (most recent call last):
  File "/private/home/anovak/workspace2/toil/venv/bin/toil-wdl-runner", line 33, in <module>
    sys.exit(load_entry_point('toil', 'console_scripts', 'toil-wdl-runner')())
  File "/private/home/anovak/workspace2/toil/src/toil/wdl/wdltoil.py", line 126, in decorated
    return decoratee(*args, **kwargs)
  File "/private/home/anovak/workspace2/toil/src/toil/wdl/wdltoil.py", line 2693, in main
    input_bindings = WDL.values_from_json(
  File "/private/home/anovak/workspace2/toil/venv/lib/python3.10/site-packages/WDL/__init__.py", line 267, in values_from_json
    raise Error.InputError("unknown input/output: " + key) from None
WDL.Error.InputError: unknown input/output: hprc_DeepPolisher.PHARAOH.alignmentPaf.threadCount

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1485

[adamnovak]

We should either determine definitively that you aren't allowed to override individual variable settings in input sections for tasks, or we should fix this to work. It might be that something about the way the task is imported from a workflow file that is causing trouble?

[unito-bot]

➤ Adam Novak commented:

We also might need a WDL conformance test for this.

[adamnovak]

OK, I tested this with this WDL:

version 1.0

workflow inputOverride {
    input {
        
    }
    call test_task {
    }
    output {
        Int result = test_task.result
    }
}

task test_task {
    input {
        Int value_in = 4
    }

    command <<<
        
    >>>

    output {
        Int result = value_in
    }
}

And this inputs file:

{
    "inputOverride.test_task.value_in": 7
}

With that WDL and that inputs, Cromwell, MiniWDL, and Toil all produce 7. If I change the call to:

    call test_task {
        input:
            value_in = 5
    }

Then Cromwell, MiniWDL, and Toil all complain that inputOverride.test_task.value_in isn't actually an input to this workflow.

[adamnovak]

The 1.0 spec doesn't seem to talk about this, but the 1.1 spec explicitly says that "It is an error for the user to attempt to override a call input at runtime", and adds a mechanism for allowing nested inputs when they aren't explicitly specified.

So I don't think this kind of override is meant to be allowed.