Support running tests in the Dart SDK repo with through test runner/CodeLens `Run | Debug` links

[nshahan]

When working on the tests from a package in the Dart SDK (https://github.com/dart-lang/sdk/tree/main/pkg/dev_compiler/test) I see links above the main method to run or debug but not above any of the group() or test() sections.

I recognize this isn't a typical configuration but it would be a great productivity boost if it is simple to get the links to appear. AFAIK these tests are run without running pub get first. This could be an issue with the analyzer?

cc @srawlins

[DanTup]

My understanding is that we can't run the tests in the SDK with dart test, as it triggers pub to run and breaks things. And as far as I know, there's no way to run specific tests when running on the VM directly (skipping the test runner).

If either of those assumptions are incorrect, it may be possible to support this but I don't know how to confirm. Do you know if that's the case?

[nshahan]

I've never run dart test or pub in this environment but I did just recently see some work being done to make that possible https://dart-review.googlesource.com/c/sdk/+/218180.

Are those the requirements for the Run | Debug links to appear in more places?

[DanTup]

Ooh, possibly. Though the comment says:

Allow pub get/upgrade to run on a number of packages in pkg/

If it only works for some packages, it may be hard for us to know which are safe. Eg., right now if I do:

cd pkg/analysis_server
dart test test/lsp/format_test.dart 

This creates a .dart_tool/package_config.json inside the analysis_server folder that resolves some things to pub (for example package:http resolves to the pub cache, but usually it would resolve to the third_party folder). I wouldn't like to enable this if it gives CodeLens links that - if clicked - might result in strange unexpected behaviour that's hard to track down.

@kevmoo is https://dart-review.googlesource.com/c/sdk/+/218180 the start of making all tests in the SDK runnable via dart test, or will it likely remain with some that can't? If the latter, is there a reliable way we could tell whether dart test is safe so we could enable these CodeLens (and other functionality, like the test runner)?

[DanTup]

Are those the requirements for the Run | Debug links to appear in more places?

To clarify - the reason we need to use dart test (or pub run test etc.) for those links to work, is that it's the only way we can selectively run a test. If we just run a test file with dart test/test_test.dart it will always run all tests in the file. Using dart test goes through the package:test runner that supports --name that lets us pass the filter to narrow it to the test where you clicked Run.

[kevmoo]

@kevmoo is https://dart-review.googlesource.com/c/sdk/+/218180 the start of making all tests in the SDK runnable via dart test, or will it likely remain with some that can't? If the latter, is there a reliable way we could tell whether dart test is safe so we could enable these CodeLens (and other functionality, like the test runner)?

Only some. This gets over the hump of dart test actually working. Your mileage may vary.

[nshahan]

With some additional changes to make sure all transitive dependencies are being discovered with local path overrides I can now run the tests in my package with dart test but I still only see the Run | Debug links on the main method in test files.

Is there anything else I need to do to cause them to appear on group and test?

[DanTup]

@nshahan sorry I wasn't very clear. There is a rule in the Dart-Code extension that detects the SDK and specifically disables this functionality because it's not guaranteed that running dart test won't break things. To support this, we'd need to relax that rule, but I only want to do it if it can be done reliably (so we don't show CodeLens links that will cause hard-to-track-down bugs).

I wonder if we can come up with a rule that would detect the projects that are ok to run.. for example, is it that any project that already has its own .dart_tool/package_config.json by valid to run dart test in? (@kevmoo)

Failing that, we could perhaps have a setting or something that would let you force it on (so it's on you if you enable it then click it where it will cause bad things).

WDYT?

[nshahan]

Another other idea @kevmoo and I were kicking around is to add an environment variable check in the dart cli to skip the pub get and just use the .package_config we manually update in the SDK repo. Then assuming that file represents a correct version solve we could assume it works for any package in the SDK?

[DanTup]

That sounds reasonable to me. From the IDE, I don't think a user would really expect running a test to trigger a package fetch anyway (if things are out of date, they'd be prompted when they open the project, and if things are missing they will see errors), so it would probably be reasonable for VS Code to just always set that env variable, and then we can probably completely remove the SDK limitation :-)