test: improve how filtering works

[perillo]

#19079 improved how filtering works, but there is still an issue.

The problem is that there are two test types: normal test (where the test name is a string literal) and a decltest (where the test name is an identifier).

An example, we have mem.test.endsWith and mem.decltest.minMax. This makes it pratically impossible for a normal user to run tests from a specified namespace, without first checking the source code.

Is it possible to exclude test and decltest when matching a filter, without causing problems?

Thanks.

[perillo]

I found another possible issue:

> zig test lib/std/std.zig --test-filter "mem" --zig-lib-dir lib
All 157 tests passed.
> zig test lib/std/std.zig --test-filter "mem." --zig-lib-dir lib
All 138 tests passed.
> zig test lib/std/std.zig --test-filter ".mem" --zig-lib-dir lib
All 65 tests passed.
> zig test lib/std/std.zig --test-filter ".mem." --zig-lib-dir lib
All 56 tests passed.

I expect the number of tests to be the same.

[nektro]

I would not expect the number of tests to be the same.

[star-tek-mb]

Also, i wanted to ask. If this is an intended behaviour:

test {
    std.debug.print("this is test_0\n", .{});
}

test "second test" {
    std.debug.print("this is second test\n", .{});
}

test "second test or third" {
    std.debug.print("this is second test\n", .{});
}

How to test only "second test" or "test_0". I found that no matter of --test-filter values - "test_0" will run (test_1, somerandomstring, anything).

zig test main.zig --test-filter "second test"

Will execute all 3 tests.

There is still no way of running single test, i think.

[adworacz]

Another thing to consider - if casing is important. Maybe the zig standard is that all tests are specified using lower case letters, but I'd wager that users are sometimes mixing cases.

So we either make the matching case insensitive, or optionally case insensitive, perhaps with regex support similar to /foobar/i.