[coverage] Support native assets in `test_with_coverage`

[liamappelbe]

Tests run under test_with_coverage can't seem to locate their native assets. This will become more of an issue when Dart 3.10 hits stable and people start using native assets in their packages. I'm already hitting it here: https://github.com/dart-lang/native/pull/2329

[liamappelbe]

The issue is much more specific than I originally thought. I'm not able to reproduce it in a toy example. The code_assets examples I tried work fine under test_with_coverage, including from a cold start where they have to build the asset first.

So it must be something specific about my package:objective_c migration that's broken.

[liamappelbe]

New hypothesis: if a library you're importing uses native assets, those will fail to load when running via test_with_coverage. I'll test this tomorrow.

[liamappelbe]

Nope, it's not just a matter of depending on a package with native assets. Toy examples using that pattern also don't repro the bug.

[liamappelbe]

Finally managed to get a minimal repro. When the package under test has a dev dependency on a package with native assets, it works under dart test but not test_with_coverage.

[liamappelbe]

The issue is that test_with_coverage runs tests through dart run instead of dart test. In fact I can repro this running the test through dart run test/foo_test.dart manually, without using coverage at all. Also repros if I run dart run test.

Minimal repro: https://github.com/liamappelbe/native_assets_test_dev_dep

@dcharkes Looks like this bug isn't coverage specific. Any ideas?

[dcharkes]

Thanks for the digging @liamappelbe! 🙏

Ah, that's a "feature". We only include dev dependencies for dart test, and we "optimize" the hooks we run for dart run to only include the normal dependencies.

If the dart run actually is dart run test, or dart run test_with_coverage, then it breaks.

Maybe we should simply also run the build hooks for dev dependencies. (Pattern matching on test or test_with_coverage sounds fragile.)

wdyt?

@goderbauer wdyt?

FWIW: In Flutter we run the hooks for dev deps always in Flutter debug mode and we never run them for release mode. So in Flutter it's not related to whether it's a test run or a program run. Any assets from a dev dependency that you might access via some Dart code from the expression compiler in the debugger would work. Since Dart doesn't have a release or debug mode. We'd essentially always be running Dart in Flutter's "debug mode".

[liamappelbe]

Ah ok. Defaulting to not building the dev deps in dart run makes sense I guess. But we probably need a way of overriding that, like a flag we can pass to dart run to build the dev deps. Then maybe the error message I've been hitting could detect if it's failing due to a missing dev dep asset, and suggest passing that flag to the run command.

If the dart run actually is dart run test, or dart run test_with_coverage, then it breaks

The problem isn't the dart run test_with_coverage command, it's that internally test_with_coverage runs the test via dart run instead of dart test. I think I did that for isolate lifecycle management reasons, but I don't remember the details. If I just switch the tool to do dart test, it doesn't manage the isolates correctly.

[goderbauer]

@dcharkes and I were just talking about it. We arrived at the conclusion that we should just run the dev dependency hooks also for dart runs. This would unify the experience between Flutter and Dart. If you want fast start-up you always have the option of dart build your program to avoid this.