Enable regular expression for stdout, stderr assertion

[pham-anh]

Prerequisites

• [x] Are you running the latest version?
• [x] Did you perform a cursory search?

Description

I think it would be nice if we can use regular expression in stdout and stderr assertion, for example:

tests:
    echo 'There are 30 dogs': 
        stdout:
            lines:
                1: There (is|are) \d+ dog[s]?

Specifications

• Version: 2.4
• Platform: Linux
• Subsystem: Centos

[dylanhitt]

I've put a bit of thought into this and I agree the feature is nice.

I can't quite decide on how to go about implementing it. I'm struggling with what users would expect the api to be.

We currently have the following:

exact

echo test:
  stdout:
    exactly: test

This seems to be pretty cut and dry and should remain as it is. There is a possibility to add regex capability here, but I personally think this should not be touched.

contains

echo hello world:
  stdout: hello # Default is a contains assertion

echo more output:
  stdout:
    contains:
      - more
      - output

We could simply change the contains matcher logic over to regex. For example the following holds true. (I don't believe this would cause any breaking changes but someone chime in if I'm missing one, my regex understanding is rough 😅 ).

package main

import (
	"fmt"
	"regexp"
)

func main() {
	r, _ := regexp.Compile("1")
	fmt.Println(r.MatchString("There is 1 dog? jello"))
}

output: true

lines

echo test\nline 2:
  stdout:
    lines:
      2: line 2 # asserts only the second line

lines is currently an exact match so we'd need to be careful with how this is implemented. We could simple switch the matcher over to regex matcher but this however would create a different behavior. For example the following example would validate as true where commander currently validates it as false

package main

import (
	"fmt"
	"regexp"
)

func main() {
	r, err := regexp.Compile("There is 1 dog?")
	fmt.Println(err)
	fmt.Println(r.MatchString("There is 1 dog? gello"))
}

In order to provide the same functionality we would need to prepend and append the beginning (^) and ending ($) anchors to lines before we validate. If the user is verbose with their regex and submit something like 1: ^There is 1 dog?$ we simply don't add the anchors.

I'm not sure if I'm missing anything here. I'm not the greatest with regex, but with a bit of research I think the following would do fine.

My main concern with the above is really the discoverability of this feature... Also a bit of logic but eh on the surface it seem pretty straightforward and really increase complexity.

Solution 2

Simply

echo test:
  stdout:
    regex: test

😄 ... This of course wouldn't support line by line assertions. We could, but I think that would fall into a bigger issue like "Support line assertions per matcher type".

@SimonBaeumer let me know what you think.

[SimonBaeumer]

@dylanhitt Changing the current behaviour to support regexes is not possible without breaking it as you pointed already out. Imho solution 2 sounds reasonable, so adding a regex and regexLines to support a regex over the whole and result and over a specific line.

We could also provide custom bash/awk verifier to execute custom code which fails if the status code is 1 and succeeds on 0.

[dylanhitt]

I agree a separate directive is the best approach.

How do you feel about pushing the lines to a separate issue? I think we could support it for every type of assertion without adding too much complexity.