wdl icon indicating copy to clipboard operation
wdl copied to clipboard

Make all examples usable as tests

Open jdidion opened this issue 3 years ago • 3 comments

This is a rather ambitious idea for the WDL spec, but would give us a big jump-start on creating a compliance test suite. The idea is for every piece of functionality in the SPEC to be accompanied by:

  • An executable test case
  • Example input
  • Expected output for executing the test with the example input

In practice, we would use collapsible (details) blocks, like this:

<details>
   <summary>
   Example: XYZ
      
   ```wdl
   task example {
    ...
   }
   ```
   
   </summary>
   
   <p>
   Example input:
   
  ```json
   {
     "input": 1
   }
   ```
   
   Example output:
   
  ```json
   {
      "expected output": 2
   }
   ```
   
   </p>
</details>

Which looks like this when rendered:

Example: XYZ
task example {
 ...
}

Example input:

 {
   "input": 1
 }

Example output:

 {
    "expected output": 2
 }

We would then have a script that extracts all the test cases from the SPEC and puts them into a nice directory structure, and then generates the pytest-wdl wrappers.

I think we should do this in the 1.1 spec (after it is approved of course - don't want to delay the vote for this).

jdidion avatar Dec 21 '20 15:12 jdidion

Great idea. We can run this in github CI as well to ensure a feature is truly implemented before it is merged into the spec.

rhpvorderman avatar Jan 04 '21 09:01 rhpvorderman

This is a great idea, and Kind of picks up where the Testing hackathon left off

patmagee avatar Jan 07 '21 19:01 patmagee

Here is the 1.1.1 SPEC WIP with the first example: https://github.com/openwdl/wdl/blob/version-1.1.1/versions/1.1/SPEC.md#an-example-wdl-workflow

jdidion avatar Jun 25 '21 21:06 jdidion