Unverified breakpoint while running tests

[Tregan]

trafficstars

Earlier today, I reported an issue over at the Dart-Code repository (https://github.com/Dart-Code/Dart-Code/issues/2742), because I was having issues with a breakpoint not triggering during tests. After some back and forth it looks like this is a Dart VM issue.

The issue is as follows:

• I have a file in lib/*.dart that I want to test, and during the test I want to step through so I set a breakpoint
• I have some test helpers in test/helpers/helper.dart. This includes some Mock classes, like a mocked class of the class in *.dart. This helper imports *.dart using package:.
• The tests for lib/*.dart are located in test/unit/*_test.dart. These tests need some functionality in helper.dart, so it imports that file using ../helpers/helper.dart. Because it tests *.dart it also imports that one using package:.
• When running the tests as debug, the breakpoint in *.dart is not triggered.
• When we remove the mocked class in helper.dart and run the test as debug again, it works fine.

Instructions on how the issue can be reproduced can be found here: https://github.com/Dart-Code/Dart-Code/issues/2742#issuecomment-680933566

@DanTup dove into it a bit more and found the following:

Great, thanks for the repro! I can reproduce this and if I try to set the breakpoints via Observatory I get these messages:

$ break /Users/danny/Desktop/TestProjects/repro_breakpoint_issue_2742/lib/breakpoint.dart:3
Script '/Users/danny/Desktop/TestProjects/repro_breakpoint_issue_2742/lib/breakpoint.dart' not found
$ break package:repro_breakpoint_issue_2742/breakpoint.dart:3
Script 'package:repro_breakpoint_issue_2742/breakpoint.dart' is ambiguous

I can't find a way to list the scripts in Observatory (nor is it implemented in VS Code), but I think this is the message we'd usually see if the script was loaded both ways. This seems like a VM bug to me - I think this should work, so I think it's worth raising at https://github.com/dart-lang/sdk (feel free to CC me when you do in case I can be of any help).

Thanks!

[√] Flutter (Channel stable, 1.20.2, on Microsoft Windows [Version 10.0.19041.450], locale nl-NL)
    • Flutter version 1.20.2 at C:\flutter
    • Framework revision bbfbf1770c (13 days ago), 2020-08-13 08:33:09 -0700
    • Engine revision 9d5b21729f
    • Dart version 2.9.1

[√] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at C:\Users\basro\AppData\Local\Android\sdk
    • Platform android-30, build-tools 29.0.3
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b04)
    • All Android licenses accepted.

[√] VS Code (version 1.48.2)
    • VS Code at C:\Users\basro\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.13.2

[DanTup]

@bkonyi FYI. Based on the error ("'package:repro_breakpoint_issue_2742/breakpoint.dart' is ambiguous") it seems like the library might be being loaded twice which usually occurs if imported with both package: and relative paths - however that doesn't seem to be the case here (or at least, not explicitly in the projects code).

[DanTup]

@bkonyi this same issue came up at https://github.com/Dart-Code/Dart-Code/issues/4015. In this case, it seems to be the presence of a Mock class in the file that's triggering it.

For example, create a new console app and replace the lib/foo.dart file with:

class Calculator {
  int calculate() {
    return 6 * 7; // Add a breakpoint here
  }
}

And update the test to call the method:

import 'package:dartcode_4015/dartcode_4015.dart';
import 'package:mockito/mockito.dart';
import 'package:test/test.dart';

// With this line commented out, the breakpoint will be hit when debugging the test.
// With this line uncommented (but nothing using it), the breakpoint will not be hit.
// class MockCalculator extends Mock implements Calculator {}

void main() {
  test('calculate', () {
    expect(Calculator().calculate(), 42);
  });
}

Commenting/uncommenting the noted line will cause the breakpoint to be hit or not (note: the class itself is not used). If you try to set breakpoints through Observatory, you get the same behaviour as above:

$ break /Users/danny/Desktop/dart_samples/dartcode_4015/lib/dartcode_4015.dart:3
Script '/Users/danny/Desktop/dart_samples/dartcode_4015/lib/dartcode_4015.dart' not found
$ break package:dartcode_4015/dartcode_4015.dart:3
Script 'package:dartcode_4015/dartcode_4015.dart' is ambiguous

[maks]

Oof! I just ran into this trying to debug a failing test and same thing still with Dart 2.18 from Flutter 3.3. If you are using Mocktail you can't seem to get breakpoints set on your app code classes referenced from your tests file (which includes Mocks) to be actually hit, eg. same as previously reported I set a break point on a print(), I see the print log out on debug console (In VSCode) but breakpoint never hits.

This makes debugging tests a very frustrating experience to say the least.