Allow users to parameterize test

[ShaddyDC]

Problem to solve

Oftentimes I want to test the same API endpoint with a couple different options and want to verify that they all work correctly. In an ideal world, I'd want to verify for each that the individual expected behaviour is correct, but for simplicity I'm often happy just to verify that it works without causing an error.

Proposal

I think it makes most sense to have variables file-wide rather than per entry. The test would then run over the cartesian product of all options. An example might look like this:

# Parameterized test for creating and verifying a resource

# Define the parameterized variables
type: ["book", "movie"]
status: ["available", "unavailable"]

# Test scenario
POST https://api.example.com/resources
{
    "type": "{{type}}",
    "status": "{{status}}"
}

HTTP/1.1 201
[Captures]
resource_id: jsonpath "$.id"

GET https://api.example.com/resources/{{resource_id}}

HTTP/1.1 200
{
    "id": "{{resource_id}}",
    "type": "{{type}}",
    "status": "{{status}}"
}

Extra useful would be if I could use a set of parameter instances to work together, so I can use a separate component in different places while they still all fit together. Example: type: [("archive", "zip"), ("audio", "mp3"), ("document", "pdf")], and then I can request an "archive" from an api and assert that the returned type is a "zip".

It should also be noted that care must be taken with the cartesian product as the number of tests can quickly explode. Maybe each time a random subset is tried?

Additional context and resources

Templates are already a supported feature. We can manually call hurl on a test with all variable combinations. This just unfortunately isn't great if you have a different set of variables per test and quickly bloats the invocation.

Tasks to complete

• [ ] Decide on a design
• [ ] Implement design
• [ ] Document design