Running the lsp causes my editor to rapidly consume all available memory

[redsuperbat]

Hi! Thanks for working on the typescript go project. I'm having an issue with the LSP. When i start it together with Neovim it causes Neovim to freeze and rapidly consume all memory available and crash.

I have tried installing via npm and building tsgo from source, both causes the same behaviour.

Before the preview was released I was able to build tsgo from source and run tsgo with neovim without problems so I'm guessing it's a recent change.

Below is a screenshot of neovim running with the tsgo lsp. The top process is the nvim process which spawned tsgo.

I'm starting the lsp with this command:

tsgo --lsp --stdio

Any help to resolve this would be greatly appreciated! If you need any more information from me please reach out!

[jakebailey]

If you are able to build and reproduce it, it would be helpful if you could bisect the problem (since we don't have any other info to go on from this).

[redsuperbat]

Thanks for a quick answer. I'll see what I can get out of bisecting.

Is there any way to toggle verbose logging from the LSP? My initial guess is that neovim somehow get's overloaded with diagnostics or something else which would consume a lot of memory.

[jakebailey]

If you want to hack it locally, just replace os.Stderr in the LSP main function with io.Discard.

[redsuperbat]

Alright after a quick bisect I was able to determine that this commit: https://github.com/microsoft/typescript-go/pull/806 is the bad one.

Unfortunately it's not 100% clear to me what the issue might be since it's quite the big commit. I will have a look and see if anything stands out...

EDIT:

By adding this

Out: io.MultiWriter(os.Stderr, os.Stdout),

To the lsp server options I was able to confirm that tsgo sends a humungous amount of these globPattern payloads:

{"globPattern":"<root path>/node_modules/next/dist/compiled/superstruct.ts","kind":1},

So I'm assuming it has something to do with the FileSystemWatcher 🤔

[jakebailey]

@andrewbranch

[sheetalkamat]

This is because of watching all failed lookup locations as is instead of directories for it.

[jakebailey]

Yeah, I think it's causing VS Code to freeze sometimes too:

[issacgerges]

Was just about to report this issue, I wonder if the issue is actually responding with the registrationOptions.watchers. In our repo (Slack) this response exceeds the body size limit (it gets to something like 1.3gb) and kills the server.

Each resolution (even if there's a file at that path, and even if theres some sort of ambient type declared like declare module *.pdf) adds watchers for index.js, index.jsx, index.ts, index.tsx, package.json, {file}.d.ts

This repro is just a simple script that generates 10k 0-byte pdfs and adds an import for each. 10k apparently isn't enough but I assume if we go high enough we'd hit the same crash

An example

[Trace - 6:17:00 PM] Received request 'client/registerCapability - (ts9)'.
ind": 1
                    },
                    {
                        "globPattern": "/Users/igerges/source/tsgo-rpc-crash/js/8815.pdf/index.js",
                        "kind": 1
                    },
                    {
                        "globPattern": "/Users/igerges/source/tsgo-rpc-crash/js/8815.pdf/index.jsx",
                        "kind": 1
                    },
                    {
                        "globPattern": "/Users/igerges/source/tsgo-rpc-crash/js/8815.pdf/index.ts",
                        "kind": 1
                    },
                    {
                        "globPattern": "/Users/igerges/source/tsgo-rpc-crash/js/8815.pdf/index.tsx",
                        "kind": 1
                    },
                    {
                        "globPattern": "/Users/igerges/source/tsgo-rpc-crash/js/8815.pdf/package.json",
                        "kind": 1
                    },
                    {
                        "globPattern": "/Users/igerges/source/tsgo-rpc-crash/js/8816.d.pdf.ts",
                        "kind": 1
                    },
                    {
                        "globPattern": "/Users/igerges/source/tsgo-rpc-crash/js/8816.pdf.d.ts",
                        "kind": 1
                    },
                    {
                        "globPattern": "/Users/igerges/source/tsgo-rpc-crash/js/8816.pdf.js",
                        "kind": 1
                    },

[redsuperbat]

What makes this problem interesting is that it seems like the LSP is telling the client to watch files in the node_modules folder. This seems a bit weird since those files are unlikely to change during development. Perhaps the globbing should exclude the files in the .gitignore if it exists and folders like node_modules by default?

Watching files in node_modules also explains why neovim crashes since node_modules tends to be thousands of javascript/typescript files.

EDIT:

Looking back at this, the kind: 1 indicates it's listening for created files, which would be fine for node_modules if the goal is to populate the type environment. However it seems like the glob pattern is expanded on the server side before sending it to the client. Perhaps that's the issue?

[cartond]

I made this repro repo. There are instructions on the README to generate the files to cause the crash.

From the testing on there, 20k files wasn't nearly enough, but 1M files reproduces the bug consistently (Exception has occurred: Error: Cannot create a string longer than 0x1fffffe8 characters). I double checked the consistency by having @issacgerges try this on his machine.

@redsuperbat in our (@slackhq) case, it wasn't just the node_modules. It was that an imported files (e.g. import icon_1 from '../files/img/icon_1.png';) would be treated as a module, so the module resolver made watcher globs for non-module files (icon_1.png in this example). The resulting stream looks something like this.

[cartond]

@redsuperbat I spoke too soon, and only looked into the first ~5k bytes of the buffer which was files like .pdf. Sorry!

When the file does resolve, the watcher still includes all the files that don't exist now, but could be created and change responses. For example

// thing.tsx
const Thing =....
...
export default Thing;

// typescript_file.ts
import Thing from '@slack/Thing';
...

The resolver (as mentioned above) then adds all the failed resolutions (globPattern) to watch:

../thing.d.ts
../thing.js
../thing.jsx
../thing.ts
../thing.tsx             <-- actually exists
../thing/index.d.ts
../thing/index.js
../thing/index.jsx
../thing/index.ts
../thing/index.tsx

when it knows thing.tsx does exist. I understand a change to any of those files would change how the lsp works. Was this how the non-go version of tsc worked?

Is there an option to exclude this overhead?

Alternatively, if the file is deleted, then add that list of options back in to watch.