l3build icon indicating copy to clipboard operation
l3build copied to clipboard

l3build check reports misleading if no tests defined

Open cfr42 opened this issue 7 months ago • 3 comments

I'm not sure if this is a bug. Maybe it is a request for clearer reporting.

If I run l3build check with a test file which fails to define any tests and an expectation file which also fails to define any, then l3build reports as follows:

Running checks on
  mini-test (1/1)

  All checks passed

It is true that all checks passed. It is equally true that all checks failed. So the final claim is technically correct, albeit misleading. The (1/1) is also confusing, I think, at least if you've seen this kind of output from make, where similar tallies typically indicate the number of tests rather than the number of files.

testfiles/mini-test.lvt:

\input regression-test.tex\relax
\typeout{hylo}

testfiles/mini-test.lve:

\input regression-test.tex\relax
\typeout{hello}

Could you clarify the output somehow? Some reminder that the numbers mean files rather than tests, maybe, and/or No checks failed rather than All checks passed. Insofar as I understand the code, isn't the former rather than the latter what's actually being reported here?

Failing that, you could remind people that l3build allows an empty domain, but while this is entirely logical, it is not very intuitive. There is something quite odd about the fact it makes claims such as 'All flying pigs speak French' and 'Every unicorn loves a Womble' true.

cfr42 avatar Jul 04 '24 04:07 cfr42