typescript-go icon indicating copy to clipboard operation
typescript-go copied to clipboard

[bug] `tsgo --build` uses extreme amounts of memory

Open mjames-c opened this issue 4 months ago • 19 comments

Stack trace

Unfortunately I don't have one to share as it crashed my remote dev machine (used 116GB of RSS before it did).

Image

Steps to reproduce

This is probably unique to our scale here at Canva. I attempted to build one of the TS projects in our monorepo that transitively depends upon 5k+ other TS projects. It built for ~5 mins before crashing out due to excess memory usage.

./built/local/tsgo --build ../canva/<omitted>/tsconfig.json

mjames-c avatar Aug 22 '25 01:08 mjames-c

Can you please try #1623 and see if that helps

sheetalkamat avatar Aug 22 '25 04:08 sheetalkamat

@sheetalkamat I just tried it out again with your change included -- unfortunately still seems to be broken. memory climbed back up to 100GB RSS before the OS killed the process. we could try to take a pprof memory profile beginning of next week if you'd find it useful?

Image
$ strings ./built/local/tsgo | grep -i 15ae26ccc
mod     github.com/microsoft/typescript-go      v0.0.0-20250822042943-15ae26cccfb7
build   vcs.revision=15ae26cccfb7cd863aa009d0171715721d6785e1

mjames-c avatar Aug 22 '25 05:08 mjames-c

@sheetalkamat

following up @mjames-c -- with profiles attached for one of our largest TS projects:

typescript-cpuprof.gz

typescript-memprof.gz

happy to provide some more profiles of smaller projects if that'll be useful as well.

johnnycheng0210 avatar Aug 25 '25 03:08 johnnycheng0210

@johnnycheng0210 can you see if you repro this if you run --singleThreaded

sheetalkamat avatar Aug 25 '25 18:08 sheetalkamat

Can you try #1692 to see if it helps. You can also use --maxConcurrentProjects to change how many projects can be built simultaneously and see if that is of any help

sheetalkamat avatar Sep 08 '25 22:09 sheetalkamat

@sheetalkamat We faced the same problem (even with --noEmit) of tsgo taking more memory than tsc. Using --singleThreaded resolved the problem, while still allowing us to take advantage of the performance gains. Thanks for the tip!

Edit: One additional thing that can be tuned is GOGC (see docs) and GOMEMLIMIT (see docs). We set GOGC to 0.3 and GOMEMLIMIT to something less than our CI servers' limit

hsource avatar Sep 11 '25 07:09 hsource

Can you please try out https://github.com/microsoft/typescript-go/pull/1745 to see if it helps

Thanks

sheetalkamat avatar Sep 22 '25 22:09 sheetalkamat

@sheetalkamat have been seeing this too on a mix of individual projects or large subsets (8k projects) in our internal monorepo, how can I help test the PR? Build locally?

I might have missed where to find the branch deploy options 🙈🙏

spanishpear avatar Sep 22 '25 23:09 spanishpear

You would need to build from https://github.com/microsoft/typescript-go/pull/1745 to test it out

sheetalkamat avatar Sep 22 '25 23:09 sheetalkamat

Cross-posting for reference https://github.com/microsoft/typescript-go/pull/1745#issuecomment-3322782149. Seems to still use a lot of memory but doesn't OOM anymore!

We also haven't tried out --maxConcurrentProjects but can give it a go. Is the default just the number of CPU cores on the machine?

mjames-c avatar Sep 23 '25 11:09 mjames-c

This seems to also be an issue with the LSP which doesn't support --singleThreaded. I was able to tune the GOGC/GOMEMLIMIT by modifying the VS Code Extension but would be nice to expose those until they have better tuning out of the box.

maxkostow avatar Dec 08 '25 18:12 maxkostow

7.0.0-dev.20251208.1

I have the same issue in our repo, we have ~1000 ts projects, building everything in single process went up to 80GB commit size, based on error messages, didn't get much further than perhaps 5%, then crashed with oom.

Running --singleThreaded works fine, everything compiles, result is still much much faster than with tsc.

Knagis avatar Dec 09 '25 14:12 Knagis

@sheetalkamat Can you resurrect your change to limit the number of concurrent project builds? We have --checkers, but I think we were also wanting to add --builders to control that number.

jakebailey avatar Dec 09 '25 16:12 jakebailey

Tomorrow, can you all try build mode with --builders 1? Given single project checking is already concurrent, we think it might be that build more concurrency may not actually matter too much anymore.

jakebailey avatar Dec 10 '25 19:12 jakebailey

Is that expected to impact the LSP or should there be a separate issue for the LSP?

maxkostow avatar Dec 10 '25 19:12 maxkostow

No, that is largely unrelated and not the focus of this thread. Build mode is not a part of the language server.

#2239 is closer, but I would very much suggest filing separate issues rather than commenting on existing ones if the cases aren't the same.

jakebailey avatar Dec 10 '25 19:12 jakebailey

@jakebailey I ran the test now with 7.0.0-dev.20251212.1

--clean called before both runs.

tsgo -b everything.json --singleThreaded - 3min 30sec to complete ~1000 tsconfig projects. ~30 sec to first error printed. Max commit size ~15GB

tsgo -b everything.json --builders 1 - 5 minutes without any output, i manually killed it. max commit size ~30GB (reached early, didn't grow much afterwards). For long periods the process seems to be idle for noticeable time, then work again, then again idle (cpu and mem doesn't move).

Knagis avatar Dec 12 '25 15:12 Knagis

Is this codebase public? That definitely sounds like the opposite of how it should behave.

jakebailey avatar Dec 12 '25 15:12 jakebailey

Unfortunately, very private. Perhaps if you are willing, we could try NDA path to get access to it. Overall it is just a lot of react tsx files and jest and playwright spec files, with declaration emit only. Few symlinks between packages, but mostly uses path aliases still.

Knagis avatar Dec 12 '25 15:12 Knagis

btw, tsgo -b everything.json --builders 1 --checkers 1 have the same behavior as simply --builders 1 - still 30GB commit size and hangs.

Also the processes hang so that i can't even kill them via task manager - it shows access denied error.

taskkill /f /im tsgo.exe
ERROR: The process "tsgo.exe" with PID 14024 could not be terminated.
Reason: There is no running instance of the task.
ERROR: The process "tsgo.exe" with PID 36272 could not be terminated.
Reason: There is no running instance of the task.

Knagis avatar Dec 15 '25 08:12 Knagis