find a better blackbox assertion than the content of stdout

[dagda1]

At the moment we are checking the output of stdout in certain places as a black box assertion that whatever the subject under test is has succeeded.

e.g.

 await World.spawn(child.stdout?.waitFor('[orchestrator] running!'));

This relies on that exactly log message being written to stdout.

Any change to the log message, .eg. removing the exclamation mark and the build will break making this a bit of a brittle assertion.

We need a less brittle way of asserting expectations.

[cowboyd]

@dagda1 I think we got rid of that with #591 Does .detect() not work? Otherwise could we enhance it to have a regexp?

[dagda1]

@cowboyd detect is a shim around waitFor

  detect(text: string) {
    return World.spawn(this.waitFor(text));
  }

It makes me feel a bit uneasy that removing a character from a log message will break the build.

I've seen a lot of builds break for assertions on text although admittedly that is label text or web content.

I totally understand why it is being used but I don't think a log message is a reliable thing. Text is prone to change and if we started streaming logs then latency is introduced.

[cowboyd]

Yeah, I can see it becoming a problem down the road, but it's been pretty servicable thus far. It's also nice because it's a very "big" test in the sense that it is very black box. How could we make it more reliable?

[dagda1]

How could we make it more reliable?

@cowboyd that is definitely the question. This sort of feels like an e2e type of test. Could we query the graphql?

[cowboyd]

For me, this feels very much like the interactor question except for the console instead of the DOM.