Filtering output support

[prologic]

Hey 👋

Been using your package here to try to write some cli integration tests for my project box and am running into a few issues that I think could be solved by implementing support for filtering output with either regular expressions or glob patterns.

See for example Mercurial's test runner and the section on Filtering output

Would you consider adding this support? 🤔

[prologic]

Here is an implementation of something similar: https://github.com/echlebek/grill/blob/b16dbd1a097be186cad7e5cf840ab802e4cfaa3a/internal/grill/diff.go which we could borrow from?

[jba]

I agree that filtering support would be nice. I don't think the (glob) and (re) suffixes are the right approach, however. The biggest problem is, what happens when you do -update? The nice thing about updating is that once you're sure your changes are correct, you can update your tests in one command with new verbatim output. I can't imagine that being smart enough to preserve globs and regexps.

I think it's better to filter in code, because you can be more precise and more general. For example, https://go.googlesource.com/vuln/+/refs/heads/master/cmd/govulncheck/main_command_118_test.go#56. Maybe there is an API that would make writing those filters simpler.

Not sure I understand the relevance of the diff code.

[prologic]

Ahh I see what you mean and what you're doing. I'm writing all my tests purely in .ct files. Maybe some some specific tests with output that changes I might be better writing those ones in Go but using the cmdtest facilities to run the binary?

[prologic]

The issue I have I guess is that if I have to write normal Go code to test some parts of my CLI (becuase they produce outputs that are not predictable) I may as well write all the tests in Go and this devalues your cmdtest package and its utility hmmm 🤔

[jba]

I haven't found that it devalues the package. There is still plenty for it to do. Notice that the tests are still encoded in .ct files, as commands and responses. I just post-process the command output in one place to generalize it.

[prologic]

For this ti work well, I would have to split out my test execution into per-directory tests containing *.ct files right? And have one func TestXXX per group of tests, where some may just run with no post-processing, whilsts others may perform some post-processing?

[jba]

I don't think so, but I may not understand your situation.

Here's an example of what I'm saying: imagine you're testing a binary called cli that sometimes prints absolute paths. Your test file looks like

$ cli somefile
wrote /home/prologic/output/somefile.out

But when I run it I get /home/jba/output/somefile.out, so the test fails.

I'm suggesting that you define cli in your test setup to run the real cli, then filter the output, replacing lines like

wrote /home/prologic/output/somefile.out

with

wrote .../somefile.out

I don't think that requires that you split your tests into different directories, or that your run some tests with post-processing and some without. But if that were necessary, it might be easier to define a separate command cli-pp that does post-processing.

Again, maybe I don't understand your use case.

[prologic]

Hmm I see....

[marco-m]

@prologic in case you are not aware, https://github.com/rogpeppe/go-internal#testscript might do what you are looking for, since it supports regexp for matching the output.