Orange-OpenSource
add support for tag based test selection
Problem to solve
An issue that I'm bumping up against is that once a test suite grows, I start to break hurl tests into separate files and while --glob does permit some utility in test selection I would find it more practical to have the ability to attach tag labels to tests and then ask url to run a secondary filter on glob patterns for tests that declare that tag
Proposal
Today's hurl files are short and sweet and I like that, but the format does lack a way to attach metadata which can then be used by the tool to when orchestrating larger suites of tests
POST url
headers
body
response assertions
borrowing from the world of markdown, hurl could use the idea of front matter to attach metadata without changing the way you describe requests. this frontmatter could then be used by the tool for things like flexible test filter, ect
---
tags: foo,bar,baz
---
POST url
headers
body
response assertions
the example here is for declaring tags metadata but because the format of frontmatter is key value based you could use the same space in the future for declaring other metadata like declaring meaningful test names for output
Additional context and resources
Tasks to complete
- [ ] ...
Hi @softprops, thank you for the feedback.
I like the idea of a Frontmatter header as it is very natural in the Markdown world. We need a way to add meta informations in Hurl files and a Frontmatter header seems a good fit. Our concern is that we want also to have syntax for specifying options and variables for a whole file (an alternative way of using command line options).
Currently we can specify options for a particular request with an [Options] section:
GET https://foo.com
HTTP 200
GET https://bar.com
[Options]
location: true
verbose: true
max-redirs: 10
HTTP 200
POST https://baz.com
HTTP 201
Given this, I can see these possiblities for addressing file options and meta:
Option 1: a Frontmatter header for the whole file (options and meta) and an [Options] section for specific request options
---
author: Jc Amiel
description: Lorem ipsum dolor sit amet, consectetur adipiscing elit
tags: [security, performance, api]
options:
- location: true
- max-redirs: 10
- variable: host=foo.com
- variable: port=3343
- skip: false
---
GET https://foo.com
HTTP 200
# [Options] ovverides options for this requst
GET https://bar.com
[Options]
location: false
verbose: true
max-redirs: 10
HTTP 200
POST https://{{host}}.com:{{port}}
HTTP 201
In this option, we can add a lot of meta description in the Frontmatter header, and also options. A cons of this syntax is that the syntax of specifying options in the Frontmatter header is different fo specifying options on a particular request.
Option 2: a Frontmatter header for the whole file (meta only), an [Options] section for the file options and an [Options] section for specific request options
---
author: Jc Amiel
description: Lorem ipsum dolor sit amet, consectetur adipiscing elit
tags: [security, performance, api]
---
# A first [Options] section for specifying file options
[Options]
location: true
max-redirs: 10
variable: host=foo.com
variable: port=3343
GET https://foo.com
HTTP 200
# [Options] overrides options for this request
GET https://bar.com
[Options]
location: false
verbose: true
max-redirs: 10
HTTP 200
POST https://{{host}}.com:{{port}}
HTTP 201
The first [Options] section seems a bit of for me, I maybe prefer the first option.
Option 3: an [Options] header for the whole file (options and meta) and an [Options] section for specific request options
# A first [Options] section for specifying file options
[Options]
location: true
max-redirs: 10
variable: host=foo.com
variable: port=3343
author: Jc Amiel
description: Lorem ipsum dolor sit amet, consectetur adipiscing elit
tags: [security, performance, api]
GET https://foo.com
HTTP 200
# [Options] overrides options for this requst
GET https://bar.com
[Options]
location: false
verbose: true
max-redirs: 10
HTTP 200
POST https://{{host}}.com:{{port}}
HTTP 201
In this option, we don't implement a Frontmatter header and we use an [Options] section for specifying meta and options for a whole file. What I don't like is that the meta informations (like tags or author) don't have equivalent command line options whereas actual [Options] section is (almost) equal to command
line options.
Why not just a new section named Meta, Metadata, Summary or any other relevant word:
[Meta]
author: Jc Amiel
title: Lorem ipsum
description: Lorem ipsum dolor sit amet, consectetur adipiscing elit
tags: [security, performance, api]
[Metadata]
author: Jc Amiel
title: Lorem ipsum
description: Lorem ipsum dolor sit amet, consectetur adipiscing elit
tags: [security, performance, api]
[Summary]
author: Jc Amiel
title: Lorem ipsum
description: Lorem ipsum dolor sit amet, consectetur adipiscing elit
tags: [security, performance, api]
I find it interesting to have (request) sections at the top of the file, before any request has been defined. For me, these sections would be applied to each request in the file. I'd still like to keep section semantics to an Entry (request/response), and not to an Hurl file.
For example, the Hurl file meta (author, description, ...) is attached to the Hurl file itself, not an entry. Something that a Frontmatter header could indeed support. => Option 2
With Option 2, a Hurl file with only one [Options] section is nice also, I agree:
[Options]
location: true
max-redirs: 10
variable: host=foo.com
variable: port=3343
GET https://foo.com
HTTP 200
GET https://bar.com
HTTP 200
POST https://{{host}}.com:{{port}}
HTTP 201
And with metadata:
---
author: jc
tags: [foo, barn baz]
enabled: true
---
[Options]
location: true
max-redirs: 10
variable: host=foo.com
variable: port=3343
GET https://foo.com
HTTP 200
GET https://bar.com
HTTP 200
POST https://{{host}}.com:{{port}}
HTTP 201
I’m not actually that invested in my format suggestion so much has having support for tag filters for flexible test selecting . Any of the options above sound good to me!
On a second pass of the alternatives above I’m thinking of tags metadata as more informational and less for the execution of the requests themselves. I think of Options as a way to codify commandline options in a hurl file in a per request basis so that you don’t need to pass them as commandline arguments.
A tags declaration wouldn’t be part of the http request itself. It’s would be informational about the files requests. It wouldn’t be an override for a commandline argument. It would define a set of possible values that a command line argument could match when selecting tests to run.