wdl
wdl copied to clipboard
Make all examples usable as tests
This is a rather ambitious idea for the WDL spec, but would give us a big jump-start on creating a compliance test suite. The idea is for every piece of functionality in the SPEC to be accompanied by:
- An executable test case
- Example input
- Expected output for executing the test with the example input
In practice, we would use collapsible (details) blocks, like this:
<details>
<summary>
Example: XYZ
```wdl
task example {
...
}
```
</summary>
<p>
Example input:
```json
{
"input": 1
}
```
Example output:
```json
{
"expected output": 2
}
```
</p>
</details>
Which looks like this when rendered:
Example: XYZ
task example {
...
}
task example {
...
}
Example input:
{
"input": 1
}
Example output:
{
"expected output": 2
}
We would then have a script that extracts all the test cases from the SPEC and puts them into a nice directory structure, and then generates the pytest-wdl wrappers.
I think we should do this in the 1.1 spec (after it is approved of course - don't want to delay the vote for this).
Great idea. We can run this in github CI as well to ensure a feature is truly implemented before it is merged into the spec.
This is a great idea, and Kind of picks up where the Testing hackathon left off
Here is the 1.1.1 SPEC WIP with the first example: https://github.com/openwdl/wdl/blob/version-1.1.1/versions/1.1/SPEC.md#an-example-wdl-workflow