Cromwell fails to return struct as output!

[antonkulaga]

Cromwell cannot handle any output that is struct base. I enclose a workflow where it is very clear to see that even though both latest development versions (as well as latest releases) of Cromwell and Womtool validated and executed the workflow for some strange reason at runtime Cromwell consider that it is a Map and not a struct and crashes in the very end of execution

 QuantifiedRun quantified_run = {"run": srr, "folder": quant_folder, "quant": quant, "lib": quant_lib}

crashes_at_runtime.zip

[antonkulaga]

In the documentations to struct the syntax that is mentioned is:

Person a = {"name": "John","age": 30}

So I expect that

 QuantifiedRun quantified_run = {"run": srr, "folder": quant_folder, "quant": quant, "lib": quant_lib}

should be treated in a similar way. If in a development version you changed the syntax the error should be thrown at compiletime and it should be explained in the documentation

[aednichols]

Is this issue intended to supersede https://github.com/broadinstitute/cromwell/issues/4555?

[antonkulaga]

@aednichols kind-of: I did not have permission to reopen #4555 and I also decided to create a simplier workflow that shows the problem

[aednichols]

Cool, thanks. We will take a look Tuesday (Monday is a holiday in the USA)

[antonkulaga]

@aednichols I managed to get it working with:

    output {
        QuantifiedRun quantified_run = object {run: srr, folder: quant_folder, quant: quant, lib: quant_lib, metadata: info}
    }

However, I am very confused why it is working as object is supposed to be deprecated in development and because you have

Person a = {"name": "John","age": 30}

as a suggested syntax and you never mention that it does not work anymore. I think it is a documentation issue that you have

[aednichols]

Hi Anton - I see that you are using version development of WDL. This is not yet an officially supported version of WDL, so indeed it is not documented.

I would recommend you use WDL 1.0, which will continue to be supported and bugfixed on future versions of Cromwell, for the foreseeable future.

[aednichols]

@antonkulaga I am running your workflow locally to investigate the issue. About how long is it supposed to take?

[aednichols]

FWIW I get

Could not localize /data/indexes/salmon/Homo_sapiens -> /Users/anichols/Documents/Repos/cromwell/cromwell-executions/quant_run/da3a4f08-7496-4242-b7c3-92b2b20acb49/call-salmon/inputs/-392899885/Homo_sapiens:
	/data/indexes/salmon/Homo_sapiens doesn't exist

when running the example workflow.

[songmingl]

Hello, I am new to WDL and have met the same problem recently. I defined a struct like this:

struct Fastp {
    File reportHtml
    File reportJson
    Array[File]+ fqs
}

and try to return it as the output in a task:

output {
    Fastp out = {
        "reportHtml": "QC/fastp.html",
        "reportJson": "QC/fastp.json",
        "fqs": if flag then ["QC/clean_1.fq.gz","QC/clean_2.fq.gz"] else ["QC/clean_1.fq.gz"]
    }
}

womtool validate was applied to check the language specification and everything went fine, but finally got the error when trying to run my workflow using Cromwell. Here is part of the error report:

java.lang.UnsupportedOperationException: Cannot construct WomMapType(WomStringType,WomAnyType) with mixed types in map values: [WomString(QC/fastp.html), WomString(QC/fastp.json), ["QC/clean_1.fq.gz", "QC/clean_2.fq.gz"]]

Any solution to this problem now?

[Coppini]

Still experiencing this problem. It seems we cannot use Array[File] inside structs for now.

version development
​
workflow Test {
    input {
        String file_name = "file.txt"
        String file_contents = "teste"
    }
​
    call WriteFile {
        input:
            file_name=file_name,
            file_contents=file_contents
    }
​
    Array[File] array_file = [WriteFile.output_file, WriteFile.output_file]
​
    MultiTypeStruct test_struct = {
        "file_name" : file_name,
        "file" : WriteFile.output_file,
        "array_file" : array_file
    }
​
    output {
        MultiTypeStruct multi_type_struct_test = test_struct
    }
}
​
struct MultiTypeStruct {
    String file_name
    File file
    Array[File] array_file
}
​
task WriteFile {
    input {
        String file_name
        String file_contents
    }
​
    command <<<
        echo -e """~{file_contents}""" > ~{file_name}
    >>>
​
    runtime {
        docker: "gcr.io/google.com/cloudsdktool/cloud-sdk:330.0.0-alpine"
        preemptible: 3
    }
​
    output {
        File output_file = "~{file_name}"
    }
}

You can easily see an error happening when running a simple workflow like this. As long as you have an Array[File] inside a struct, it will keep on failing. In my case, I'm using version development, and the last task on the workflow simply gets stuck with status Running while the workflow itself moves to status Aborting and stays stuck permanently in Aborting (never actually moving its status to Aborted).

Experienced this issue with Cromwell versions 63 and 74, while using GCP lifescience v2 backend.

[AsierGonzalez]

@aednichols I managed to get it working with:

    output {
        QuantifiedRun quantified_run = object {run: srr, folder: quant_folder, quant: quant, lib: quant_lib, metadata: info}
    }

However, I am very confused why it is working as object is supposed to be deprecated in development and because you have

Person a = {"name": "John","age": 30}

as a suggested syntax and you never mention that it does not work anymore. I think it is a documentation issue that you have

This also worked for me. I had issues with a workflow that tried to use this struct:

struct SampleAndUnmappedBams {
  String base_file_name
  Array[File] flowcell_unmapped_bams
  String unmapped_bam_suffix
}

This wouldn't work:

SampleAndUnmappedBams alignment_analysis_struct = {
    "base_file_name": sequencing_metadata.readgroup_name,
    "flowcell_unmapped_bams": [ Preprocessing.output_unmapped_bam ],
    "unmapped_bam_suffix": "unmapped.bam"
  }

But changing it to this did the trick:

SampleAndUnmappedBams alignment_analysis_struct = object {
    base_file_name: sequencing_metadata.readgroup_name,
    flowcell_unmapped_bams: [ Preprocessing.output_unmapped_bam ],
    unmapped_bam_suffix: "unmapped.bam"
  }

@Coppini, you can see that I am using an array of files in the struct and it worked, so at least that is solved now if it was really the issue.