Optimize analysis speed for large projects

[Giuspepe]

Running the custom lint analysis slows down disproportionately for large projects. This is a huge impediment for using custom_lint on large projects.

I have made some measurements on a very small project and a large project with one custom linter which yields a lint for each file and one custom linter which yields nothing at all. There was no significant difference between the two custom linters. As a baseline, I used dart analyze. The measurements were conducted on a MacBook Pro (M1 Max, 32 GB RAM).

	baseline (dart analyze)	flutter pub run custom_lint
small project	3 seconds	17 seconds
large project	11 seconds	753 seconds

The small project was created with flutter create small_project and thus only has 2 dart files with 145 lines of code. The large project has 3285 dart files with 282901 lines of code.

It would be great if the performance of custom_lint could be improved to come close to the performance of dart analyze. The current performance issues make custom_lint practically unusable on large projects, even though large projects are the ones that would benefit the most from using custom_lint. I'd be glad to help out too if you provide me with some guidance 😊

Output of flutter doctor -v

[✓] Flutter (Channel stable, 3.0.4, on macOS 12.2.1 21D62 darwin-arm, locale en-DE)
    • Flutter version 3.0.4 at /Users/giuseppecianci/fvm/versions/stable
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 85684f9300 (12 days ago), 2022-06-30 13:22:47 -0700
    • Engine revision 6ba2af10bb
    • Dart version 2.17.5
    • DevTools version 2.12.2

[!] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/giuseppecianci/Library/Android/sdk
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.

[✓] Xcode - develop for iOS and macOS (Xcode 13.3.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] VS Code (version 1.69.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.44.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-arm64   • macOS 12.2.1 21D62 darwin-arm
    • Chrome (web)    • chrome • web-javascript • Google Chrome 103.0.5060.114

[✓] HTTP Host Availability
    • All required HTTP hosts are available

[rrousselGit]

What plugin are you using to come up with this result?

[Giuspepe]

I used the sample linter plugin from the documentation:

import 'dart:isolate';

import 'package:analyzer/dart/analysis/results.dart';
import 'package:custom_lint_builder/custom_lint_builder.dart';

// This is the entrypoint of our custom linter
void main(List<String> args, SendPort sendPort) {
  startPlugin(sendPort, _ExampleLinter());
}

// This class is the one that will analyze Dart files and return lints
class _ExampleLinter extends PluginBase {
  @override
  Stream<Lint> getLints(ResolvedUnitResult resolvedUnitResult) async* {
    // A basic lint that shows at the top of the file.
    yield Lint(
      code: 'my_custom_lint_code',
      message: 'This is the description of our custom lint',
      // Where your lint will appear within the Dart file.
      // The following code will make appear at the top of the file (offset 0),
      // and be 10 characters long.
      location: resolvedUnitResult.lintLocationFromOffset(0, length: 10),
    );
  }
}

I also ran the measurements with a linter which yields nothing, but there was no significant difference.

The time was measured with the time command (time dart analyze and time flutter pub run custom_lint).

[rrousselGit]

Interesting. I'll look into it

Do you have the same result in the IDE? It could be unique to the command line.

[Giuspepe]

It's actually faster in the IDE than on the command line. Still a bit slower in the large project compared to the small project, but it's not that big of a difference like when comparing them on the CLI.

I found one of the causes of the performance issue. I'm using fvm to quickly switch between Flutter versions. This is achieved by a relative symlink at <path to my flutter project>/.fvm/flutter_sdk which links to the cache of my selected Flutter version. In the logs of custom_lint I noticed that my custom linter is also analyzing the files linked by fvm - which is the whole Flutter SDK.

After removing the .fvm folder, flutter pub run custom_lint only took 59 seconds to run. This is better, but there's still room to improve when compared to dart analyze which takes 11 seconds in the same project.

dart analyze works in combination with fvm, so I think that custom_lint should do so too without requiring extra configuration by the user. Do you think it'd be a good idea to implement a fix by not following relative symlinks?

[rrousselGit]

Ah, fmv is an interesting consideration. I've never used it, hence why I didn't come across this issue. I'll think about what can be done.

[Giuspepe]

I've noticed that custom_lint analyzes code imported by plugins as well, e.g. ios/.symlinks/plugins/flutter_secure_storage/example/lib/main.dart. I think custom_lint shouldn't do this because dart analyze doesn't do that either. This also contributes to the slower performance when comparing custom_lint to dart analyze.

[rrousselGit]

Thanks for the info. I'll try looking into it next week

[kuhnroyal]

I can confirm that the custom lint is checking .fvm/flutter_sdk and ios/.symlinks folders. Maybe by default all hidden folders should be skipped. It would also be great if the ignores from the analysis_options.yaml can be taken into account as well, I have a lot of generated files ignored there (drift, freezed etc.).

[TimWhiting]

Can one of you who uses fvm try it out on this PR? https://github.com/invertase/dart_custom_lint/pull/33

[Giuspepe]

I just tried it and it works great with fvm @TimWhiting. With your changes it now only takes 44 s to run flutter pub run custom_lint in my large project. Thanks for your contribution!

[TimWhiting]

You can also watch #31 for more some followup speed improvements, and of course there is always seeing if there is something you are doing that is inefficient in the custom linter you are creating as well. Good luck!

[rrousselGit]

With #33 closed, this probably can be closed too.