Meson and Perl disagree on UNEXPECTEDPASS TAP test point failing the test file

[nmisch]

https://testanything.org/tap-specification.html#todo-tests says "Should a todo test point begin succeeding, the harness should report it as a bonus." Meson chooses to treat that as a form of failure. The Perl modules in which TAP originated, however, treat it as success. To see this, run prove --verbose unexpected_pass.pl, with unexpected_pass.pl containing:

use Test::More;
ok(1, 'not TODO');
Test::More->builder->todo_start('demo');
ok(1, 'UNEXPECTEDPASS');
ok(0, 'EXPECTEDFAIL');
Test::More->builder->todo_end;
done_testing();

The return code is zero, and the output is:

unexpected_pass.pl .. 
ok 1 - not TODO
ok 2 - UNEXPECTEDPASS # TODO demo
not ok 3 - EXPECTEDFAIL # TODO demo
#   Failed (TODO) test 'EXPECTEDFAIL'
#   at unexpected_pass.pl line 6.
1..3
ok
All tests successful.

Test Summary Report
-------------------
unexpected_pass.pl (Wstat: 0 Tests: 3 Failed: 0)
  TODO passed:   2
Files=1, Tests=3,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.02 cusr  0.00 csys =  0.03 CPU)
Result: PASS

I see Meson chose intentionally (search for "unexpected" and "todo"): https://github.com/mesonbuild/meson/pull/4832 https://github.com/mesonbuild/meson/issues/2923 https://github.com/mesonbuild/meson/pull/12362

I'm filing this issue mostly to help other people find it. That said, we could change the documentation or eliminate the difference.

[nmisch]

Applications needing Perl's interpretation can wrap the TAP source like in https://github.com/postgres/postgres/commit/22a4b10

[bonzini]

Closing as intentional behavior.

[eli-schwartz]

https://www.gnu.org/software/automake/manual/html_node/Generalities-about-Testing.html

Automake treats TODO tests as errors as well. The TAP spec's wording says "treat it as a bonus" but doesn't really mandate whether a "bonus" means the results collector should return overall success or overall failure. It's "enough" to simply list it in a different column.

At least I'd expect the default behavior to remain the same (it is very useful IMO) with, if anything, a meson test flag to avoid erroring out. I suppose a documentation update would be fine, too.