[coverage] Pub workspace support
We should start supporting pub workspaces in package coverage.
Intended workflow: dart pub global run coverage:test_with_coverage in a workspace directory, it should produce a coverage file spanning all sources of all packages in that workspace. Lines covered by the tests of another package should also count as coverage.
@liamappelbe Things that need to change:
- The file filtering
- Multiple different coverage runs for combining
Since we don't have package:test support for workspaces, the way to run all the workspace tests in a single command at the moment is to manually specify all the test directories: dart test pkgs/foo/test pkgs/bar/test. This works as long as the tests are written to not assume a particular working directory.
Something similar almost works for coverage: dart run coverage:test_with_coverage -- pkgs/foo/test pkgs/bar/test. This works except that the scoping is broken, causing the output to be empty. Currently, if you don't specify --scope-output, it defaults to the package name. For workspace packages that name won't work as a filter (it's typically just _). If you modify the command to specify the scopes, it works: dart run coverage:test_with_coverage --scope-output foo --scope-output bar -- pkgs/foo/test pkgs/bar/test.
So I'll definitely need to update the defaulting behavior of --scope-output. That will require parsing the pubspec.yaml file. Then dart run coverage:test_with_coverage -- pkgs/foo/test pkgs/bar/test will work. Workspaces can be nested, so I'll need to parse the pubspecs recursively.
To get the minimal dart run coverage:test_with_coverage working automagically, I have two options:
- Wait for
package testsupport, so thatdart testin the workspace directory runs all the subproject tests. - Since I have to parse the pubspecs anyway, I already know all the subdirs that need to be passed to the
dart testcommand. Not sure if that's a good idea though, since it assumes the tests don't care about their working dir. In fact, I have a lot of tests where this won't work. It might be confusing for users if we try to do this magic thing, and their tests start failing in strange ways.
I'm inclined to go with option 1, so for now I'll just fix --scope-output. This will mean dart run coverage:test_with_coverage -- pkgs/foo/test pkgs/bar/test will work, but dart run coverage:test_with_coverage won't for now.
This works as long as the tests are written to not assume a particular working directory.
Yeah, I've been fixing tests so that they don't assume this. This enables running all tests in one swoop which makes the CI much faster for mono-repos. However, I don't think this is something the ecosystem has set on this.
This will mean
dart run coverage:test_with_coverage -- pkgs/foo/test pkgs/bar/testwill work, butdart run coverage:test_with_coveragewon't for now.
SGTM, I can easily make that work (https://github.com/dart-lang/native/pull/2316).
The next step is seeing how coveralls (and codecov) will fare with paths being both reported from the repo root and from package roots. 🙃