Specification
Specification copied to clipboard
Extending TAP to define parameters controlling the TAP producers.
Hi,
I am looking at TAP from the point of view of IDE integration. TAP currently defines a way for test runners to report testing results to consumers, hence IDE being a consumer can reuse the reporting parsing code, while using test runners interchangeably. However, there are 2 other common features that IDEs implement:
- enumerating all tests without running them (happens when user opens workspace)
- running selected subset of tests(happens when user selects a single test and runs)
Currently IDEs would have to implement this logic for each test runner, even if the runner itself is a TAP producer. This makes it impossible to make a universal TAP test adapter for an IDE and just use TAP-compatible runners interchangeably.
My idea is currently that TAP can have a small extension, defining 2 CLI parameters:
- --dry-run to produce the TAP output just skipping every test.
- --tests
only run tests whose description matches the provided glob.
Please let me know if what I am describing makes sense.
The idea of TAP is a universal output, or result. The idea is that any "TAP consumer" can parse what a "TAP produce" spits out. Defining CLI arguments for producers is out of scope. I think this is a step that would be a mistake to take. This is additionally true if we consider existing TAP producers, which are usually harnesses that usually already have --dry-run or similar flags, and may or may not already implement --tests.
We do not want to start enforcing arguments on harnesses/producers
Agreed. Out of scope for the protocol specification, this type of behavior is left for producers.