dart-lang
How to run an excluded test?
I have decorated my test file with @Tags(['integration']) at the top
I then created a dart_test.yaml file with exclude_tags: integration
https://github.com/dart-lang/test/blob/master/pkgs/test/doc/configuration.md#include_tags
If I run all my tests my integration test doesn't run - as expected.
How do I run that test? I don't want to accidentally run other integration tests so think commenting out the tags or dart_test.yaml code would be error prone.
You need to pass that tag explicitly -t integration. If you want to only run that integration test, make sure to also pass the path to that test explicitly. So dart test -t integration test/my_integration_test.dart.
I'm still not able to get this to override the dart_test.yaml exclude.
My dart_test.yaml:
exclude_tags: integration
My command
dart test -t integration test/blah_test.dart
Tag at top of test
@Tags(['integration'])
Output with the dart_test.yaml exclude tags specified
No tests ran.
No tests match the requested tag selectors:
include: "integration"
exclude: "integration"
Output without tag
00:01 +1: All tests passed!
Maybe the test is being excluded and the exclude overrides the include??? I want to override the exclude but maybe that is not possible???
Apologies @jakemac53 for delay, I was on an older version of Dart and I wanted to upgrade. Dart - version 2.18.2 Test - 1.21.6
Hmm yeah, it is possible the excludes override the includes. The precedence gets pretty weird here, because probably what you really want is for command line includes to not be overridden. And then having overlapping includes/excludes in the dart.yml file would be an error? I don't know that we really have good infra for supporting that right now, and all the merging of these args is already quite complicated as it stands...
yup, it seems like it doesn't do what I want it to do. Maybe we need a exclude_tags_unless_specified item in the test_dart.yaml ??? I guess I could create a saved test for each project that excludes those 'integration' tests but it would be nice to do it on a project level (or even global).
Maybe you could also try setting up two different presets instead to work around this. So the exclude config wouldn't exist at all in the other preset.
Yes, I think the design intention here was that you'd create two presets.
I do think it would be reasonable for -t command line arguments to take precedence, but I don't know how complicated that would be to implement.
We merge the configuration from the CLI arguments with the configuration from dart_test.yaml too early to differentiate between the difference sources for includes. In order to implement this feature I think we'd need to add a separate commandLineArgsInclude field. I'm not eager to increase the complexity of using Configuration and SuiteConfiguration, so I'd like to understand the level of pain it causes to not have it.
@atreeon - can you try adding separate local and integration presets to your dart_test.yaml and let use know how well that works for you?
My pain is not too severe thankfully but I think it has at times stopped me writing some testing which is possibly a problem.
Thinking some more now with the extra info you've kindly provided, it might be more of the interface between IDEA and the test commands where my specific problem is.
Idea uses the following commands:
Right click the test & run
dart run test -r json myprojectdir/test/calculator_test.dart -N "subtraction"
Right clicking on a file & run:
dart run test -r json myprojectdir/test/calculator_test.dart
Right clicking on the project & running all my tests uses the command
dart run test -r json myprojectdir
I use this right clicking and running my tests a lot while I'm developing.
When my 'integration' tag is on the individual test and I run the test file or test suit as a whole, I want that test to be excluded but if I ran the individual test I want that specific test to run.
Similarly if the tag is on the individual file and I ran the test file I would expect it to run but excluded from when I run all my tests.
I can set some presets up and I think they will get me some way there and I'll get by but I think the above is kind of sensible. Saying that it is the interface may not be exactly true as I'd probably want / expect the same from the command too.
That sounds like more of an issue for DartCode then, it would have to support passing different args somehow. But I am really not sure what that would look like.
cc @DanTup in case you have any ideas around using test profiles in the plugin. Maybe a given test or file could be associated with a profile somehow?
@atreeon mentioned IDEA above, so I think may be using IntelliJ? I can't speak for that, but for VS Code:
If the goal is to run the test when invoking that test specifically, this doesn't sound very different to the skip flag. When you run a single test from VS Code, we always pass the --run-skipped flag to tell package:test to run it regardless of skip flag. Would it make sense to have something similar here (--run-excluded?) that can be used when running an individual test so that it's always run?
You can also customise the arguments passed to tests in VS Code with args in a launch configuration in launch.json. You can also contribute custom CodeLens links and/or override the default Run/Debug behaviour. There's an example here which will cause all runs of tests to include -p "chrome,vm" --test-randomize-ordering-seed=12345 whether they're run from Test Runner, Run menu or CodeLens:
https://dartcode.org/releases/v3-34/#debugger
That said, package:tests's presets don't look all that different to VS Code's test run "profiles". It could be nice if we can auto-discover those and populate them so they can be run from the test runner/gutter icons. I don't know how well it'd work (I don't know much about either, and when I last looked, I think profiles were intended to be workspace-wide, not project-wide), but it might be worth some investigation. Although, if we start integrating this better (which I think we probably should), we should consider what we can/can't rely on from Dart-Code to avoid breaking things in future.
