Implement clarifications and new conformance tests for 1.2.1

[mr-c]

requests to fix any of the below should go against https://github.com/common-workflow-language/cwl-v1.2/tree/1.2.1_proposed You can preview all the implemented changes for v1.2.1 so far but know that they have not been ratified by the CWL Leadership Team; additional comments are requested.

Anyone is welcome to take one of these and start adding the needed conformance tests and specification clarifications! Please just drop a note in the linked issue when you start.

If you have never written a CWL conformance test or edited the text of the CWL standards before, that's okay! Just ask for instructions on what to do in the linked issue of your choice.

Candidates issues

Rough consensus with a known solution

New conformance tests

See https://github.com/common-workflow-language/cwl-v1.2/blob/main/CONFORMANCE_TESTS.md#format-of-the-conformance-test-file for information on how to write a new conformance test

ToDo

• [ ] Conformance tests for renaming output files with basename
  • [ ] confirm that fixing the above also covers everything in Clarify the description of basename modification in the spec
• [ ] Add tests for itemSeparator
• [ ] Conformance test for optional record input
• [ ] need conformance test for workflow output + linkMerge: merge_flattened
• [ ] There are no specific conformance tests about the behavior for nested inputBinding
• [ ] Add conformance test for ECMAScript 5.1 strict mode
• [ ] create conformance test for InitialWorkDirRequirement w/ & w/o Docker and reference to input file paths
• [ ] Continue with Audit the objects available for every place we allow parameter references/expressions producing new conformance tests for each category
• [ ] new conformance test: symlinks using absolute paths in a Directory
• [ ] two directory inputs with same basename but different paths
• [ ] dockerLoad, dockerFile, and dockerImport lack conformance tests
• [ ] workflow in subdirectory, tools in another subdirectory, run contains relative paths like run: ../tools/my_tool.cwl. Example; source: https://github.com/DataBiosphere/toil/pull/4052#issuecomment-1149705565
• [ ] test for parameter references in CommandLineTool.arguments (pull request)
• ~~a proper test for WorkReuse~~ (not possible with cwltest)
• [ ] potential secondaryFiles conformance test candidate (cannot reproduce with cwltool)

Specification clarification

changes to wording of the specification, may require new conformance test(s) to clarify or confirm

ToDo

• [ ] Specification is unclear on whether array typeDefs are allowed in SchemaDefRequirement
• [ ] Document that Directory loadListing behavior in v1.1 is not backwards compatible
• [ ] Clarify records with multiple inputBindings
• [ ] Several terms are used for the value of record type: objects, maps, and records
• [ ] Define decoding method for loadContents and cwl.output.json
• [ ] clarify use of parameter references / expressions in ResourceRequirement fields
• [ ] Does CommandLineBinding.loadContents apply to secondary files?
• [ ] Expected behavior of outputBinding on array of file
• [ ] be explicit about how to format an input object file via the optional cwl-runner CLI
• [ ] Improve explanations about array level and per-item inputBinding
• [ ] better document the 'default' and 'valueFrom' fields on WorkflowStepInput
• [ ] clarify type checking rules
• [ ] WorkflowOutputParameter outputSource should not be optional
• [ ] Examples for linkMerge merge_flattened
• [ ] specify YAML version, which schema
• [ ] fields not specified in an input object get a value of 'null'
• [ ] clarify sort order for globs
• [ ] clarify LoadListingEnum
• [ ] [prose] Confirm that final output objects must have the listings of all directories recursively expanded
• [ ] explain how to render a type: enum on the command line
• [ ] clarify evaluation policy of Workflow level requirements
• [ ] Explain the concept of a Abstract CWL Workflow, a class: Workflow with one or more class: Operation steps (or sub-steps). An Abstract CWL Workflow description is not (fully) executable, as Operation processes are themselves abstract; but it does allow describing a workflow and providing a skeleton to attach metadata about the steps, authorship, tools used, and more. Abstract CWL Workflows are used in the RO-Crate workflows and workflow runs profiles; the former used by workflowhub.eu.

Bonus issues

These are worthy, but we don't have a solution ready or obvious plan yet

Pending Discussion:

• [ ] directory with executable script inside → https://github.com/common-workflow-language/cwl-v1.2/pull/263

ToDo

• [ ] JS expression returning Undefined
• [ ] document format schema bridging technique
• [ ] Definition of hints is inconsistent
• [ ] Clarify, make explicit, and add conformance test if CommandLineTool requires a baseCommand vs being a no-op
• [ ] default values with secondaryfiles defined by extension needs a conformance test
• [ ] "Edit This Section" button for the rendered CWL standards

Done

• [x] outputEval: "time passed" syntax error
• [x] Allow [] ? in types under ArraySchema.list
• [x] conformance tests: add docker.io/ to dockerPull entries that lack an explicit registry
• [x] clarify $(..) vs ${...}
• [x] conformance test requires location: Any what does it mean ? (in progress at https://github.com/common-workflow-language/cwl-v1.2/pull/124)
• [x] testing InitialWorkDirRequirement with directories as output and more (in progress at https://github.com/common-workflow-language/cwl-v1.2/pull/126 )
• [x] Conflict between the conformance test 61 and the spec for parameter references (in progress at https://github.com/common-workflow-language/cwl-v1.2/pull/64)
• [x] new conformance test: $(runtime.outdir) for outputBinding glob https://github.com/common-workflow-language/cwl-v1.2/pull/164/
• [x] Unclear specification of run field in the workflow description
• [x] mention cwl-upgrader; explain version mixing https://github.com/common-workflow-language/cwl-v1.2/pull/172
• [x] fix the broken InitialWorkDirRequirement link
• [x] remove the initial_workdir_output_glob_duplicate test common-workflow-language/common-workflow-language#160
• [x] The ID numbers are off by one at 220 or before Irrelvant due to common-workflow-language/cwltest#110
• [x] conformance test for input record + JS https://github.com/common-workflow-language/cwl-v1.2/pull/170
• [x] New SchemaDefRequirement conformance test candidate https://github.com/common-workflow-language/cwl-v1.2/pull/171
• [x] conformance test needed: MultipleInputFeatureRequirement on workflow outputs https://github.com/common-workflow-language/cwl-v1.2/pull/169
• [x] Define the concept of an "opaque string" https://github.com/common-workflow-language/cwl-v1.2/pull/162
• [x] affirm that WorkflowOutputParameter.outputSource can reference workflow level inputs https://github.com/common-workflow-language/cwl-v1.2/pull/178
• [x] Stdout behavior with chained commands https://github.com/common-workflow-language/cwl-v1.2/pull/190
• [x] clarify meaning of Process.id https://github.com/common-workflow-language/cwl-v1.2/pull/166
• [x] Why conformance test 63 expect empty result ? https://github.com/common-workflow-language/cwl-v1.2/pull/184
• [x] https://github.com/common-workflow-language/cwl-v1.2/issues/193
• [x] #239
• [x] Prose for location in File/Directory mentions URIs instead of IRIs
• [x] About specification when output class is File
• [x] common-workflow-language/common-workflow-language#232

[cwl-bot]

This issue has been mentioned on Common Workflow Language Discourse. There might be relevant details there:

https://cwl.discourse.group/t/dockerfile-contents-vs-dockerfile-file/221/6

[smyja]

ItemSeparator test has been written already.

[mr-c]

ItemSeparator test has been written already.

Hello @Smyja ; if you think https://github.com/common-workflow-language/common-workflow-language/issues/834 has been fixed, you should comment in that issue. However, I don't see the referenced tests merged into this repository, just simple usage of itemSeparator in two other tests