What does this mean?

[BJWiley233]

I ran code just to test converting CWL pipeline below:

java -jar dxCompiler-2.4.7.jar compile /Users/****/****/UKBB/analysis-workflows/definitions/pipelines/CH_exome_Final2.cwl -language CWL -project UKBB_Exome_2021 -folder /CH_Exome/

And I get this error. What do I have to do/create/make/add to clear this error? Any advice is welcomed.

[error] Error creating translator for /Users/****/****/UKBB/analysis-workflows/definitions/pipelines/CH_exome_Final2.cwl
java.lang.RuntimeException: missing definition for schema file:///Users/****/****/UKBB/analysis-workflows/definitions/types/labelled_file.yml#labelled_file
	at dx.cwl.CwlType$.inner$1(CwlType.scala:117)
	at dx.cwl.CwlType$.translate(CwlType.scala:137)
	at dx.cwl.CwlArray$.translate(CwlType.scala:451)
	at dx.cwl.CwlSchema$.translateSchema(CwlType.scala:326)
	at dx.cwl.CwlType$.inner$1(CwlType.scala:96)
	at dx.cwl.CwlType$.translate(CwlType.scala:137)
	at dx.cwl.CwlType$.apply(CwlType.scala:144)
	at dx.cwl.CommandInputParameter$.apply(CommandLineTool.scala:65)
	at dx.cwl.CommandLineTool$.$anonfun$apply$19(CommandLineTool.scala:202)
	at scala.collection.Iterator$$anon$9.next(Iterator.scala:575)
	at scala.collection.mutable.Growable.addAll(Growable.scala:62)
	at scala.collection.mutable.Growable.addAll$(Growable.scala:59)
	at scala.collection.mutable.ArrayBuffer.addAll(ArrayBuffer.scala:147)
	at scala.collection.mutable.ArrayBuffer.addAll(ArrayBuffer.scala:40)
	at scala.collection.mutable.ArrayBuffer$.from(ArrayBuffer.scala:265)
	at scala.collection.mutable.ArrayBuffer$.from(ArrayBuffer.scala:253)
	at scala.collection.IterableOps.map(Iterable.scala:671)
	at scala.collection.IterableOps.map$(Iterable.scala:671)
	at scala.collection.AbstractIterable.map(Iterable.scala:920)
	at dx.cwl.CommandLineTool$.apply(CommandLineTool.scala:201)
	at dx.cwl.Parser.parse(Parser.scala:64)
	at dx.cwl.Parser.parseFile(Parser.scala:92)
	at dx.translator.cwl.CwlTranslatorFactory.create(CwlTranslator.scala:160)
	at dx.translator.TranslatorFactory$$anonfun$createTranslator$6.applyOrElse(Translator.scala:97)
	at dx.translator.TranslatorFactory$$anonfun$createTranslator$6.applyOrElse(Translator.scala:97)
	at scala.collection.IterableOnceOps.collectFirst(IterableOnce.scala:1086)
	at scala.collection.IterableOnceOps.collectFirst$(IterableOnce.scala:1078)
	at scala.collection.AbstractIterable.collectFirst(Iterable.scala:920)
	at dx.translator.TranslatorFactory$.createTranslator(Translator.scala:97)
	at dxCompiler.Main$.compile(Main.scala:394)
	at dxCompiler.Main$.dispatchCommand(Main.scala:765)
	at dxCompiler.Main$.main(Main.scala:870)
	at dxCompiler.MainApp$.delayedEndpoint$dxCompiler$MainApp$1(Main.scala:875)
	at dxCompiler.MainApp$delayedInit$body.apply(Main.scala:874)
	at scala.Function0.apply$mcV$sp(Function0.scala:39)
	at scala.Function0.apply$mcV$sp$(Function0.scala:39)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
	at scala.App.$anonfun$main$1(App.scala:73)
	at scala.App.$anonfun$main$1$adapted(App.scala:73)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:920)
	at scala.App.main(App.scala:73)
	at scala.App.main$(App.scala:71)
	at dxCompiler.MainApp$.main(Main.scala:874)
	at dxCompiler.MainApp.main(Main.scala)

[jdidion]

As stated in the readme, CWL support is under active development. You can consider it "alpha" right now. Your pipeline likely uses some features that are not yet supported.

[BJWiley233]

Thanks, what stage is WDL on? And also what maybe the best resource to look at for directly creating DNAnexus applet workflows using scripting and not through the interface? Thanks for your help.

[emiloslavsky]

WDL is ready for use. Is the example at https://github.com/dnanexus/dxCompiler/#example-workflow enough to get you started?

[BJWiley233]

Yes I know WDL, I was referring to writing straight DNAnexus workflow applets using coding, i.e. skip the WDL or CWL conversion step.

[oklempir-cf]

xVantage Care Support team sent the following response via ticket tracking system (SF27964):

Based on our investigation, it appears that your CWL workflow is designed for CWL 1.0. And as it has been already mentioned here (https://github.com/dnanexus/dxCompiler/issues/149) - CWL 1.2 support are under active development.

"WDL draft-2 and versions 1.0 and 1.1 are fully supported, while WDL 2.0 (aka 'development') and CWL 1.2 support are under active development." [https://github.com/dnanexus/dxCompiler/]

As our dxCompiler-2.4.7.jar testing example, we were able to successfully build a CWL workflow (written in version 1.2.):

https://github.com/common-workflow-language/cwl-v1.2/blob/main/tests/dynresreq-workflow-inputdefault.cwl

using the following command

java -jar dxCompiler-2.4.7.jar compile /Users/oklempir-cf/Downloads/cwl-v1.2-main/tests/dynresreq-workflow-inputdefault.cwl -project SF27964 -folder /cwl_test/

In order to convert your code from 1.0 to 1.2, you may find the following conversion/normalizer tool useful (it is created by the main developer of CWL language Michael R. Crusoe).

https://github.com/common-workflow-language/cwl-utils

Please note that this is an untested suggestion and we have not tested this conversion tool on any workflow available at https://github.com/genome/analysis-workflows/

If you wish to write your custom straight DNAnexus workflow applets using coding, we recommend using the following doc page on how to start writing custom made app(let)s. There are several ways how to implement your applets, e.g. with Python or Bash:

https://documentation.dnanexus.com/developer/apps

https://documentation.dnanexus.com/developer/apps/intro-to-building-apps

[BJWiley233]

So I updated my pipeline to v1.2 and still get same issue missing definition for schema. Does this have to do with update to SchemaDefRequirement? Since my issue fails with all of these types:

requirements:
- class: SchemaDefRequirement
  types:
  - $import: ../types/labelled_file.yml
  - $import: ../types/sequence_data.yml
  - $import: ../types/trimming_options.yml
  - $import: ../types/vep_custom_annotation.yml

We have the four types here: https://github.com/genome/analysis-workflows/tree/master/definitions/types

Can you test if dxCompiler works for a schemadef as in: https://github.com/common-workflow-language/cwl-v1.2/blob/main/tests/anon_enum_inside_array_inside_schemadef.cwl

UPDATE, this anon_enum_inside_array_inside_schemadef.cwl worked so I am must be missing something for custom types

[BJWiley233]

So I figured out what the issue is. As in the spec for CWL v1.2 you are allowed to list import types and use them as arrays with the syntax:

type: labelled_file.yml#labelled_file[]

where #labelled_file represents the name of the type in the imported .yml file but the dxCompiler thinks the entire string labelled_file.yml#labelled_file[] is the type when in CWL the # symbols for the name of the input.

See https://www.commonwl.org/v1.2/Workflow.html#SchemaDefRequirement and here is example (type: biom-convert-table.yaml#table_type) of v1.0 but this has not changed for v1.2

Additionally answer see here on biostars: https://www.biostars.org/p/259322/

[commandlinegirl]

Thanks @BJWiley233, we've been working on a stable version of CWL (v2.10) and have fixed a couple of issues related parsing of identifiers. We'll incorporate the example you posted into our test suite and check if it this can be considered fixed.