`test.py` doesn't report failure in case of `info`-level issue and `--fatal-infos`

[sgrekhov]

trafficstars

I'd expect that the following tests should be treated as failing by the test.py.

// SharedOptions=--fatal-infos

@deprecated
String variable = "x";

main() {
  print(variable);
}

This test is reported as paiing. But we have here info-level issue (use of a deprecated variable) and --fatal-infos analyzer option. With this option enabled analyzer's return code became 1 and test.py should report it as a failure. But it doesn't.

[dart-github-bot]

Summary: The test.py script fails to report a test as failing when an info-level issue is encountered with the --fatal-infos flag enabled. This is because the analyzer returns a non-zero exit code in this scenario, which should be interpreted as a test failure by test.py.

[lrhn]

If you check the code of pkg/test_runner/lib/src/test_file.dart, it has a comment saying:

/// *   Flags can be passed to dart2js, VM or dartdevc by adding a comment to
///     the test file:
///
///         // SharedOptions=--flag1 --flag2
///

That at least suggests that the analyzer won't see the flags.

I'd have thought that they would, because experiment flags work, but experiments are parsed out separately from the SharedOptions lines, so it's quite possible that the analyzer sees only those, and not the remaining shared options.