fortran-lang
Support test anything protocol (TAP) in fpm test
Test anything protocol: https://testanything.org/
It would be nice if fpm test could provide a TAP consumer when running the testsuite. Having to pipe the fpm test output through a TAP consumer or have a TAP consumer as --runner is quite error prone in case somebody forgets to add those as TAP tests always exit with error code zero.
Fpm could detect TAP automatically or require to enable it on per test basis with:
[[test]]
name = "tap-producer"
tap = true
Can you clarify how would TAP be detected automatically?
I presume the test main program would import the module definitions with use tap (or use fpm_tap if we could implement our own TAP layer).
In any case I like the succinctness of tap = true.
The TAP output is quite distinct, we could capture the standard output of the test and check for the first line starting with 1.., which indicates that we are using TAP. Capturing standard output shouldn't introduce a delay, this has to be designed rather carefully.
That sounds reasonable, but I don't think the interest in TAP among all FPM users will be big enough to make it enabled by default. The explicit approach sounds better to me.
A compromise could be to have a project-wide setting auto-tap = true that would turn on automatic TAP consumption for all tests. Tests which are not TAP would then need to be disabled explicitly with tap = false. But I fear this is already getting too confusing.
This is an interesting idea. I would expect fpm would be a TAP consumer, but maybe it should just support TAP consumers as plugins? I also like the idea of being explicit about which tests provide TAP output, as that would be much easier than trying to guess based on its output.
I read the TAP specification, and there are some complexities that would encourage us not to try and reinvent our own parser, not least of which being that it would need to include a yaml parser. Also, the version specifier at the start is optional, and the plan line may appear at the start of output, the end of output, or not at all, so it's not sufficient for determining whether the output is TAP.
If this is something that moves forward, I might be interested in developing a vegetables spin-off that is (at least initially) API compatible with the current vegetables, but produces TAP output. The only thing I don't like about TAP is that it doesn't seem to provide any nice way of organizing tests in a hierarchical way, but a test suite that is organized hierarchically can be flattened without too much difficulty.
