test_runner: support glob inclusions and exclusions

[avivkeller]

This PR adds the functionality to exclude test files by glob patterns.

It is meant for the use case when a user may have hundreds of test files and may only want to run a few (via glob).

Added test results:

▶ --test-runner-include
  ✔ No globs (86.421831ms)
  ✔ *.test.js (82.534169ms)
  ✔ nonexistent.js (44.967071ms)
▶ --test-runner-include (215.595732ms)
▶ --test-runner-exclude
  ✔ No globs (86.749583ms)
  ✔ o*.test.js, i*.test.js (87.369228ms)
  ✔ ** (48.384698ms)
▶ --test-runner-exclude (223.456528ms)
ℹ tests 8
ℹ suites 0
ℹ pass 8
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 442.787698

[nodejs-github-bot]

Review requested:

• [ ] @nodejs/test_runner

[github-actions[bot]]

The https://github.com/nodejs/node/labels/notable-change label has been added by @RedYetiDev.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

[avivkeller]

src/node_options.cc:881:  Small and focused functions are preferred: EnvironmentOptionsParser::EnvironmentOptionsParser() has 506 non-comment lines (error triggered by exceeding 500 lines).  [readability/fn_size] [1]

I'm not sure what to do about this? This might cause an issue for any additions to the CLI flags.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 59.25926% with 11 lines in your changes missing coverage. Please review.

Project coverage is 87.32%. Comparing base (a7271ab) to head (92c484b). Report is 65 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/test_runner/runner.js	47.61%	11 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #54600      +/-   ##
==========================================
- Coverage   87.33%   87.32%   -0.01%     
==========================================
  Files         649      649              
  Lines      182570   182595      +25     
  Branches    35040    35030      -10     
==========================================
+ Hits       159454   159458       +4     
- Misses      16390    16404      +14     
- Partials     6726     6733       +7     

Files with missing lines	Coverage Δ
lib/internal/test_runner/utils.js	64.12% <100.00%> (+0.27%)	:arrow_up:
src/node_options.cc	87.91% <100.00%> (-0.12%)	:arrow_down:
src/node_options.h	98.21% <ø> (ø)
lib/internal/test_runner/runner.js	87.25% <47.61%> (-1.17%)	:arrow_down:

... and 30 files with indirect coverage changes

[meyfa]

I'm not sure about the -runner infix. No other flag seems to be using it.

Other than that, this feature seems very useful 👍

[avivkeller]

I'm not sure about the -runner infix. No other flag seems to be using it.

I figured that because we have --test-coverage-[include/exclude], it would make sense to use --test-runner- as a prefix.

[mcollina]

How does this affect run()?

[cjihrig]

Maybe I'm missing something, but I don't understand what the purpose of --test-runner-include is. Isn't this the same functionality as passing globs on the command line currently?

--test-runner-exclude makes more sense, but I'm not sure I see much value in it. Has anyone actually requested this functionality?

I'm not sure the extra CLI flags are worth it.

[avivkeller]

How does this affect run()?

I haven't yet integrated this into the run() function, but I will if you'd like

Maybe I'm missing something, but I don't understand what the purpose of --test-runner-include is. Isn't this the same functionality as passing globs on the command line currently?

If the user would like to include multiple different globs, they can with this flag.

--test-runner-exclude makes more sense, but I'm not sure I see much value in it. Has anyone actually requested this functionality?

I haven't seen any requests for this in the issue tracker, but I would assume that it is the gradual "next step" for --test-[name/pattern]-include (but for entire files).

[cjihrig]

If the user would like to include multiple different globs, they can with this flag.

Isn't that already possible though?

$ node --test 'test1/**/*.js' 'test2/**/*.js'
✔ test 1 (0.735125ms)
✔ test 2 (0.730708ms)
ℹ tests 2
ℹ suites 0
ℹ pass 2
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 52.44975

[avivkeller]

Oh... I didn't know that 😅. However the -exclude may still be favorable in some cases, but if not, I'm happy to close this.

[cjihrig]

I won't block it if other people find it useful, but I'd rather not add it at this time.

[cjihrig]

@RedYetiDev what are you planning to do with this PR? If it's not going to land, it would be good to close.

[avivkeller]

I'll close it for now, sorry for the trouble