Completion slow in large projects

[ghostdogpr]

Describe the bug

I noticed that code completion speed seem proportional to the size of the project. On small projects it is really fast but on large projects it can be really slow (up to 5-6 seconds in my case).

Here's the same example inside a small project vs a large project:

https://github.com/user-attachments/assets/8e4ed41f-d270-4467-b73f-6f07406ab63f

https://github.com/user-attachments/assets/87805848-1a6b-49ec-9d7b-cc687fe94862

It's the exact same code that doesn't depend on anything else but you can see how the completion is instant in one case but takes a few seconds in the other one.

Is there anything that can be done about it?

Expected behavior

Fast completions.

Operating system

macOS

Editor/Extension

VS Code

Version of Metals

1.5.2

Extra context or search terms

Using sbt as my build server.

[tgodzik]

Thanks for reporting! It would probably be great to run async profiler for those cases and send the results. I noticed that it usually is a case of a size of a file, not of a project so this is surprising.

What is the Scala version you are using?

[ghostdogpr]

Thanks for reporting! It would probably be great to run async profiler for those cases and send the results. I noticed that it usually is a case of a size of a file, not of a project so this is surprising.

What do I need to profile exactly? Metals itself?

Yeah it is surprising to me as well, in this case it's a very small file that I just created and it's still affected by the rest of the project.

What is the Scala version you are using?

3.6.4

[tgodzik]

What do I need to profile exactly? Metals itself?

Yes, the presentation compiler is running inside.

3.6.4

There is a fix that should be available in 3.7.1-RC1 that might help, but I am not 100% sure.

[ghostdogpr]

[EDIT: attached some flamegraph files below]

[ghostdogpr]

There's a lot of time spent in dotty.tools.dotc.classpath.AggregateClassPath.list(String) AggregateClassPath.scala:24 which leads to dotty.tools.dotc.classpath.DirectoryLookup.list$$anonfun$1(Function1, PackageName, Function1, Object) DirectoryClassPath.scala 5646 243 and many file operations.

[tgodzik]

Yeah, that's usually the likely culprit. Not sure why the compiler needs so many file operations. It looks like inverseSemanticdbSymbol is the biggest cause here.

[ghostdogpr]

Here's a couple flamegraphs from async profiler:

On my large file where it takes 5-6 seconds: flamegraph.html.zip

On the small file from the video: flamegraph2.html.zip

[pluveto]

It's really slow, sometimes I cannot even connect to the LSP because it crashes. Code formatting can take up to 1 minute to finish... It ate all my memory(64gb) when I was using OpenJDK and then I swithed to GraalVM which saved me 50% mem.

[tgodzik]

It's really slow, sometimes I cannot even connect to the LSP because it crashes. Code formatting can take up to 1 minute to finish... It ate all my memory(64gb) when I was using OpenJDK and then I swithed to GraalVM which saved me 50% mem.

Looks like a different issue, though I haven't seen problems like yours. If you could try with async profiler for formatting(this should not take a minute) and maybe provide a bit more information ina separate issue.

This issue is strictly about completions and the most likely reason is that we are reading too many files. Your problem seem that the compilation takes up a lot of time and Bloop is using a lot of memory.

[tmiseikis]

I'm also experiencing this issue. It takes 10+ seconds on my large projects for code completion to appear even if completion is on the class defined in the same file:

https://github.com/user-attachments/assets/93ab51a8-a631-44a9-aac6-0433a6b746d6

Flamegraph: flamegraph.html.zip

[tgodzik]

@tmiseikis what version are you using? I think 3.7.1 should include some improvements (3.3.7 which will be released together with 3.7.3)

[JakobLyngPetersen]

I experience the same slowness. Also in 3.7.1. I have attached a video showing that it takes around 4 seconds from I press Ctrl+Space until I see the result. The file is a short file, but it's in a project with around 11000 Scala files and about 1.2 million lines of code.

https://github.com/user-attachments/assets/01aa2cd6-15a4-4b2a-bfee-5464305d2858

flamegraph_3.7.1.html.gz

[tgodzik]

Thanks! This at least gets rid of the last problem, now it seems SemanticdbSymbols.tryMember is causing most of the issues. I guess it accesses the codebase classpath, which is why this is happening.

It's looking for extension methods there.

[JakobLyngPetersen]

Thanks! Extension methods are obviously very relevant. Would it make sense to show the results gradually? Something like showing "normal matches" as soon as those have been resolved, then adding-in extension methods as they get resolved? I realize that this might lead to confusing behavior ("jumping" items or items that are not shown in the usual sorting order) so it may be a bad idea. Just a thought. The best would, of course, be that it's just always "fast enough" 🙂.

[tgodzik]

I think the worst case is with no prefix, which tries to find every extension method. We can do some catches for sure to improve.

[tmiseikis]

@tgodzik - In my original video I used Scala 3.4.2 and Metals 1.5.3. After bumping Scala version to 3.7.1 and updating Metals to 1.6.0 it looks like the code completion got even slower. It takes almost 20 s. now for code completion to show up in a large project.

https://github.com/user-attachments/assets/0a378dd9-820c-4579-8df1-196d8e360040

Flamegraph:

flamegraph-3.7.1.html.gz

[tgodzik]

I think I see at least two issues, one should be simple to solve, the other less so. Is this also happening if you add anything after the dot? Like myObject.m

[tmiseikis]

@tgodzik - no, typing anything after the dot is fast.

[tgodzik]

Ok, I think I should be able to fix it by 3.7.2 and 3.3.7. you should be able to test a nightly next week