mesonbuild
Check stdout/stderr from test run
I would like to verify that a program fails with a specific error message when presented with specific input.
The obvious solution would be to wrap program execution in another script that verifies the expected output, but this does not combine well with the --wrap option -- e.g. with valgrind, I'd like to analyze my program, not the test tool. In addition, having a generic mechanism would be easier for portability.
Use case: I have a compiler that should report syntax errors with a location. The test should verify that an ill-formed input is both rejected (should_fail: true) and outputs the correct location for the error.
For me, exact match is probably sufficient, but I'd expect a new wishlist bug to pop up asking for regex support immediately.
Well, I'm somewhat concerned about adding regex support since there are quite a few variants of regex. Unfortunately the one python provides isn't guaranteed to be consistent with what's available in other languages, so if we guarantee meson users that they can use python-compatible regular expressions, this would present an undue burden to rewriting meson in a language other than python.
It's a real issue for muon. /cc @annacrombie
Matching on (a substring of) exact text should be much easier.
Indeed.
The other thing I'm slightly concerned about is wrapper behaviour: if my expectation is that stderr is 2:2: invalid character, and the test runs with valgrind, then I probably also get the entire valgrind output to stderr -- so I can't use an exact match anymore even though the output from the tool is exact.
I think this kind of matching tends to get complicated quickly, which is why the common solution is to use a wrapper or to use some kind of diff against a known-good output.
With respect to --wrap, as you say there is no right answer - doing valgrind outside the matching wrapper means testing the matcher; but doing the matching directly in Meson would mean including extra output from the valgrind stderr and possibly causing failures.
I'm not sure how to proceed here.
