microsoft
[lsp][bug] LSP locks up for ~12s when navigating to files within the same project
Apologies if we're reporting this too early but we're excited to use the typescript-go LSP at Canva when it's ready.
We have a TS project with 90k+ files within it. The initial load time for the project is very impressive at ~20s (vs 2.5 mins with TSv5):
Info 8 [09:49:07.603] Project '<omitted>/tsconfig.json' (KindConfigured)
Files (92699)
The problem we have is that navigating the code base causes the LSP to lock up for ~13s. According to the LSP logs it is dominated by getConfigFileNameForFile (5s) and updateGraph (7s).
Info 39 [09:52:20.374] getConfigFileNameForFile:: File: <omitted> ProjectRootPath: :: Result: <omitted>/tsconfig.json
Info 40 [09:52:25.344] Config: <omitted>/tsconfig.json : {
...
Info 41 [09:52:26.557] Starting updateGraph: Project: <omitted>/tsconfig.json
Info 42 [09:52:32.612] Finishing updateGraph: Project: <omitted>/tsconfig.json version: 5
Info 43 [09:52:32.612] Different program with same set of files
Info 44 [09:52:32.627] Project '<omitted>/tsconfig.json' (KindConfigured)
Files (92699)
I assume there's a caching layer that still needs to be ported to tsgo that already exists TSv5.
sharing some additional investigation i did today -- i manually wrapped the handleDidOpen() function (LSP command textDocument/didOpen) with the pprof profiler and took a profile of an incremental update (opening a previously unopened TS file within the same project).
tsgo --lsp seems to be spending most of its time stat'ing files:
Duration: 19.29s, Total samples = 71.61s (371.23%)
Showing nodes accounting for 71.61s, 100% of 71.61s total
FYI we released a preview extension: https://marketplace.visualstudio.com/items?itemName=TypeScriptTeam.native-preview
I am curious if this still happens; a lot of stuff has been optimized. Maybe not this, since we haven't stuck FS caching in the LSP yet.
@jakebailey just tried it out now (Identifier: typescriptteam.native-preview, Version: 0.20250601.1)
at a high level the problem still persists, here are a subset of the logs after using go-to def on a symbol within the same project:
Info 17 [10:00:17.762] Starting updateGraph: Project: <omitted>/web/tsconfig.json
Info 18 [10:00:17.762] Program 2 used 1 checker(s)
Info 19 [10:00:29.053] Different program with same set of root files
Info 20 [10:00:44.289] createGlobMapper took 6.702289805s to create 8 globs for 2733778 failed lookups
Info 21 [10:00:44.293] createGlobMapper took 4.250189ms to create 5 globs for 1750 failed lookups
Info 22 [10:00:44.293] Finishing updateGraph: Project: <omitted>/web/tsconfig.json version: 2 in 26.531445762s
Info 23 [10:00:44.293] Project '<omitted>/web/tsconfig.json' (KindConfigured)
Files (98150)
Info 26 [10:00:50.829] Starting updateGraph: Project: <omitted>/web/tsconfig.json
Info 27 [10:00:50.829] Program 3 used 2 checker(s)
Info 28 [10:01:01.016] Different program with same set of root files
Info 29 [10:01:16.437] createGlobMapper took 6.803180171s to create 8 globs for 2733778 failed lookups
Info 30 [10:01:16.441] createGlobMapper took 4.148414ms to create 5 globs for 1750 failed lookups
Info 31 [10:01:16.441] Finishing updateGraph: Project: <omitted>/web/tsconfig.json version: 3 in 25.611635744s
Info 32 [10:01:16.441] Project '<omitted>/web/tsconfig.json' (KindConfigured)
Files (98150)
The update graph now takes ~26s and, interestingly, it seems to run twice -- once when i first initiate the go-to def and then again once VS Code opens the file with the symbol definition.
It's probably also worth pointing out that the tsgo LSP uses 22.6 GiB of memory (vs ~17ish for TSv5) -- the initial startup updateGraph seems to be better though taking ~37s in the tsgo LSP vs ~2m15s in TSv5.
Let me know if another pprof profile on handleDidOpen() would be helpful 🙂
here's an updated pprof profile from handleDidOpen() -- the critical path seems to be dominated by updating file watchers (~15s) and visiting directories as part of tryFindDefaultConfiguredProjectForOpenScriptInfo() (5s).
naively, both of these things sound like they should be skipped when the user is performing actions which don't mutate FS state
It's possible https://github.com/microsoft/typescript-go/pull/996 will help, but it does clear the cache between changes, so maybe not, actually.
We just aren't doing any sort of caching when it comes to module resolution, which means no caching of globs either.
I added back the cache for config file for open file in https://github.com/microsoft/typescript-go/pull/1007 and that might help with tryFindDefaultConfiguredProjectForOpenScriptInfo
Also we dont have proper program reuse code (its on TODO) and that means if you open and close file, it would mean program update for now. (because the text on disk could be different - optimization that is not yet done)
tried this out again recently and it seems to be much improved! go-to def cross-file navigations are now on the order of ~4s for our mega project.
Info 28 [09:28:45.646] getConfigFileNameForFile:: File: <omitted> ProjectRootPath: :: Result: <omitted>/tsconfig.json
Info 29 [09:28:48.147] MemoryStats:
Alloc: 13294817 KB
Sys: 16352340 KB
NumGC: 78
Info 30 [09:28:48.147]
Project '<omitted>/tsconfig.json' (KindConfigured)
Files (98175)
-----------------------------------------------
Open files:
FileName: <omitted>ProjectRootPath:
Projects: <omitted>/tsconfig.json
FileName: <omitted> ProjectRootPath:
Projects: <omitted>/tsconfig.json
FileName: <omitted> ProjectRootPath:
Projects: <omitted>/tsconfig.json
-----------------------------------------------
Info 31 [09:28:49.191] getConfigFileNameForFile:: File: <omitted> ProjectRootPath: :: Result: <omitted>/tsconfig.json
that said would be great if they were instant instead 😁
closing this out as we don't really experience this anymore at Canva. unrelated to original ticket but we've seen over 10x+ speed gains for intellisense loading for one of our biggest projects! thanks again for all the hard work.
@mjames-c That’s great to hear! I noticed that your June 17 comment where go-to-def was taking 4 seconds was before the snapshot LSP refactor, but either way, 4 seconds seems long for go-to-def. Has that changed since?
I noticed that your June 17 comment where go-to-def was taking 4 seconds was before the snapshot LSP refactor, but either way, 4 seconds seems long for go-to-def. Has that changed since?
@andrewbranch yep, it seems to be much snappier now! to add a little more clarity we have two project structures we've been testing tsgo lsp on (with a total TypeScript code base size of ~115k files):
- a very fine grained project (references) structure (30k+ tsconfig.json) and
- a monoproject structure (
tsconfig.all.json) which encapsulates all ~115k files
the ~4s go-to def wait time I mentioned before was in reference to the tsconfig.all.json approach. this largely seems to be gone.
In case it's interesting to ya'll when testing on tsconfig.all.json the project load time is ~34s on a 64 GB memory, 8 vCPU machine. The LSP consumes ~19.8 GiB of RES memory in this case.
The feature that we're next most looking forward to having is a work find-all refs and rename https://github.com/microsoft/typescript-go/issues/1219 😁