allow ignoring tests

[amtoine]

trafficstars

i have a test in nu-git-manager that cannot run for now and thus i thought it would be cool to ignore it.

Description

for now, the only way to ignore a test is to not export it. however, by doing so, the test won't appear in the output of nupm test...

i think it would be cool to support something like

# tests/mod.nu
export def my-test-ignore [] {
    # ...
}

and still have my-test appear in yellow next to the other ones :yum:

Testing package /path/to/my-package
tests test-fail-1 ... FAIL
tests test-fail-3 ... FAIL
tests test-success ... SUCCESS
tests test-ignore-1 ... IGNORE
tests test-fail-2 ... FAIL
tests test-ignore-2 ... IGNORE
Ran 5 tests. 1 succeeded, 2 ignored, 3 failed.

[NonlinearFruit]

A big draw back to using the test name is if you have a test like ignore-node-modules or similar would be skipped

But definitely a way to skip tests without modifying them would be awesome

[vyadh]

I didn't actually see what nupm was doing to run tests (I initially assumed it was written in Rust) but I see it's doing similar to my recent PR to the documentation that illustrates a tiny embedded test runner method that does something very similar.

That ignores tests like this:

# ignore
def test_fib_ignored_test [] {
  print "This test will not be executed"
}

by doing

 scope commands
      | where ($it.name | str starts-with "test_")
      | where not ($it.description | str starts-with "ignore")
     # ...

I imagine there is a whole bunch of stuff that could be driven from description instructions. E.g. running a test method with different test data.

[Mrfiregem]

With attributes I think this would be pretty ergonomic. Here's a proof-of-concept implementation by filtering on tests annotated with @category 'ignore'.

[132ikl]

it would probably be better to use an attribute specifically for this use case instead of repurposing category. Ex:

export alias "attr ignore" = echo

@ignore
export def ignore-test [] {
    assert equal true false
}

[Mrfiregem]

Yeah, I mentioned that in the commit message. There's the extra overhead of defining the command in the tests module, but it definitely reads better.