go-langserver uses lot of cpu and this causes vscode to hang or slow responsiveness

[chandradeepak]

Macbook OS x EI Captain

Attached is the cpu profile . It shows lot of time spent in reading files not sure why that happens but i have seen this slows the find references in vscode very slow. I have almost 50 files in a project.

let me know if you need any thing else

[chandradeepak]

cpu.svg.zip

[keegancsmith]

Thanks for the report @chandradeepak as well as the profile. How does the resource usage compare to the reference finder without the language server? Our reference implementation is a modified version of the reference finder in go guru (what the plugin uses).

When did you install the language server? We did some major updates a few months ago which should help a lot with resource usage, so just in case you have an old lang server binary hanging around I'd suggest updating.

[keegancsmith]

The trace seems to point to most of the time spent reading files. Right now the implementation relies on the go typechecker. It shouldn't do more work than go does to compile your application. IS the time take a similar amount of time without a pkg cache?

[chandradeepak]

what is a pkg cache? Not sure what it is. I know my language server is latest it was installed on Nov 6th 2017.

[keegancsmith]

Sorry, I missed your reply. @chandradeepak if you still have this issue, how does the perf compare to just vscode-go without the langserver? You say you have 50 files, how many files are on your GOPATH though?

[chandradeepak]

@keegancsmith , vscode-go would be smooth until i change some files with out langserver . But it has some other issues as well. In my GOPATH i have so many repos. Do you think that is the cause ?

[slimsag]

@chandradeepak To clarify what @keegancsmith said above a bit: find references with the language server is more slow if your code imports a lot of Go packages, or if the packages you import themselves import a lot of Go packages (i.e. transitive dependencies). The more Go files your program ultimately uses in the end, the slower Find References is in general.

If you disable the language server in vscode-go settings and just use vscode-go without language server, how slow is find references in comparison?

Note: We do have someone who is taking a look at improving find references performance, but it's a very tough problem to solve and as such there is no ETA on when it may be completed right now.

[danicat]

Is anyone exploring this at this moment?

[slimsag]

In order to fix this issue, we will need to switch over to using the new golang.org/x/tools/go/packages package. Almost all of the CPU and memory usage from go-langserver comes from type-checking, and using that package would reduce it substantially as it would cache results to disk automatically AFAIK.

Nobody is working on this issue right now AFAIK, and it'd probably be quite a large undertaking as it would affect almost the entire codebase I think -- but we'd be very open to someone trying to contribute this. 😃I can't give any guarantees or hard dates, but I think we are likely to work on this in a few months or so if nobody beats us to it.

[danicat]

Looking at the documentation on go/packages it says it's not production ready and has dependencies on Go 1.11. The idea is to wait until it gets stable?

[slimsag]

The warning about it not being ready for widespread use yet seems to be mostly about the fact that the API may change, which is OK for us (we can vendor it anyway).

If we use it before it supports older Go versions, we'd need to use it optionally at build time. e.g. when building with Go 1.11, it would use that package and when using an older version it would use our current go/loader setup.

[kelbyers]

I am experiencing high CPU usage from this recently. I have vscode up and running in the background, on a different desktop, and go-langserver shows up as using 50% of a CPU on my Mac in the activity monitor.

[primalmotion]

Mine goes as high as 400% every 10m. It's not usable.

[akzk]

The go-langserver uses 800% cpu (8 cores) and 15GB mem (16GB total) on my MBP. It happens in this month, I do not know why it happens

[keegancsmith]

Can you share your config? Have you enabled any optional features like code completion or diagnostics?

[akzk]

I write Go file by vscode, here is my settings.json

{
    "workbench.editor.enablePreview": false,
    "files.autoSave": "off",
    "editor.fontSize": 13,
    "go.testFlags": ["-v", "-count=1"],
    "go.testEnvVars": {"MY_DEVBOX_HOST": "10.8.122.122"},
    "go.docsTool": "gogetdoc",
    "go.goroot": "/usr/local/Cellar/go/1.10.3/libexec",
    "go.gopath": "/Users/leonardo/Documents/go",
    "go.gocodeAutoBuild": false,
    "go.formatTool": "goreturns",
    "go.useLanguageServer": true,
    "C_Cpp.intelliSenseEngine": "Default",
    "python.pythonPath": "python3",
    "python.formatting.provider": "yapf",
    "code-runner.clearPreviousOutput": true,
    "code-runner.saveFileBeforeRun": true,
    "code-runner.executorMap": {
        "python": "env LC_ALL=en_US.UTF-8 python3",
        "cpp": "cd $dir && g++ -std=c++11 $fileName -o $fileNameWithoutExt -w && $dir$fileNameWithoutExt && rm $dir$fileNameWithoutExt"
    },
    "window.zoomLevel": 0,
    "extensions.ignoreRecommendations": false,
    "explorer.confirmDelete": false,
    "git.autofetch": true,
    "git.confirmSync": false,
    "terminal.integrated.shell.osx": "/bin/zsh",
    "gitlens.advanced.messages": {
        "suppressShowKeyBindingsNotice": true
    }
}

BTW, there are 1281 Go files in my project, too much files ？

[keegancsmith]

You are not opting into anything that should make go-langserver that slow. 1281 files should be fine. We run go-langserver on sourcegraph.com and don't use as much resources for large projects such as kubernetes. We won't have time to investigate these perf issues in the short-term, so I would suggest disabling the go language server in vscode-go in the mean time. There should hopefully be a new go language server in the not too distant future which won't suffer from such issues. See https://github.com/sourcegraph/go-langserver/issues/316#issuecomment-429307592

[sp4rd4]

I have the same issue, it goes almost up to 600% cpu usage, I have almost 2000 files in a project. Here is config:

{
    "[markdown]": {
        "editor.acceptSuggestionOnEnter": "off",
        "editor.formatOnSave": true,
        "editor.renderWhitespace": "all",
        "editor.wordWrap": "on"
    },
    "bracketPairColorizer.colorMode": "Consecutive",
    "bracketPairColorizer.showBracketsInGutter": true,
    "bracketPairColorizer.showHorizontalScopeLine": false,
    "bracketPairColorizer.showVerticalScopeLine": false,
    "editor.fontFamily": "Source Code Pro",
    "editor.fontSize": 13,
    "editor.formatOnPaste": true,
    "editor.insertSpaces": false,
    "editor.lineHeight": 18,
    "editor.minimap.enabled": false,
    "editor.multiCursorModifier": "ctrlCmd",
    "editor.rulers": [
        120,
        80
    ],
    "editor.scrollBeyondLastLine": false,
    "editor.snippetSuggestions": "top",
    "editor.tabSize": 2,
    "explorer.confirmDelete": false,
    "explorer.confirmDragAndDrop": false,
    "explorer.openEditors.visible": 0,
    "extensions.ignoreRecommendations": false,
    "files.exclude": {
        "**/.git": true,
        "**/.vscode": true,
        "**/bower_components": true,
        "**/node_modules": true,
        "**/vendor": true
    },
    "files.trimTrailingWhitespace": true,
    "gitlens.advanced.messages": {
        "suppressCommitHasNoPreviousCommitWarning": false,
        "suppressCommitNotFoundWarning": false,
        "suppressFileNotUnderSourceControlWarning": false,
        "suppressGitVersionWarning": false,
        "suppressLineUncommittedWarning": false,
        "suppressNoRepositoryWarning": false,
        "suppressResultsExplorerNotice": true,
        "suppressShowKeyBindingsNotice": true,
        "suppressUpdateNotice": true,
        "suppressWelcomeNotice": false
    },
    "gitlens.codeLens.enabled": false,
    "gitlens.historyExplorer.enabled": true,
    "gitlens.keymap": "alternate",
    "gitlens.statusBar.enabled": false,
    "go.autocompleteUnimportedPackages": true,
    "go.lintFlags": [
        "--config=~/.gometalinter"
    ],
    "go.lintTool": "gometalinter",
    "go.useLanguageServer": true,
    "search.exclude": {
        "**/.git": true,
        "**/.vscode": true,
        "**/bower_components": true,
        "**/node_modules": true,
        "**/vendor": true
    },
    "search.location": "panel",
    "search.useIgnoreFiles": false,
    "sync.autoDownload": false,
    "sync.autoUpload": false,
    "sync.forceDownload": false,
    "sync.quietSync": false,
    "sync.removeExtensions": true,
    "sync.syncExtensions": true,
    "telemetry.enableCrashReporter": false,
    "telemetry.enableTelemetry": false,
    "window.zoomLevel": 0.4,
    "workbench.activityBar.visible": false,
    "workbench.colorTheme": "Material Theme Darker High Contrast",
    "workbench.editor.enablePreview": false,
    "workbench.iconTheme": "eq-material-theme-icons-darker",
    "workbench.startupEditor": "newUntitledFile"
}

[keegancsmith]

@Sp4rd4 for now just disable the language server. If your performance improves report back here.

[sandipb]

I ran dtruss on go-langserver to see why it is using so much CPU. I saw that it was churning over and over the Go source files in GOROOT. I moved the Go src files to a different place, and the CPU usage became ok. Of course this means that I can't get doc info for standard library stuff, but I can atleast go through my own code. I cannot disable language server, it is the only alternative which currently supports symlinks in GOPATH, and that it critical for me.

I wish the langserver would read GOROOT/src only once during startup and not watch those directories afterwards. The number of open/fstat64 going on was ridiculous.

[keegancsmith]

@sandipb thanks, that is a good lead on what is going wrong and surprising to me. If you re-install the langserver using the same version of Go you are developing with do you still have issues?

[sandipb]

I have configured vscode to compile go-langserver using the executable for the project in a version specific tools directory. So it is compiled with the same version.

Just to confirm, I used dlv as instructed by https://dave.cheney.net/2017/06/20/how-to-find-out-which-go-version-built-your-binary

Confirmed that it is the same version.

(dlv) p runtime.buildVersion
"go1.11"

[primalmotion]

It seems I managed to calm go-langserver down by dropping the vendor directory and building it with whatever I had in my GOPATH. It's been an hour now and it seems to behave.

I'll update if it goes crazy again

[primalmotion]

It's not better. It also caused a kernel panic

*** Panic Report ***
panic(cpu 6 caller 0xffffff80114b9dc3): "compressed PTE 0xffffff92279aba98 0xc000000000000000 has extra bits 0x8000000000000000: corrupted?"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4903.221.2/osfmk/i386/pmap_x86_common.c:1192
Backtrace (CPU 6), Frame : Return Address
0xffffff92279ab7d0 : 0xffffff80113aca9d 
0xffffff92279ab820 : 0xffffff80114e6893 
0xffffff92279ab860 : 0xffffff80114d82ba 
0xffffff92279ab8d0 : 0xffffff8011359ca0 
0xffffff92279ab8f0 : 0xffffff80113ac4b7 
0xffffff92279aba10 : 0xffffff80113ac303 
0xffffff92279aba80 : 0xffffff80114b9dc3 
0xffffff92279abb60 : 0xffffff80114bade7 
0xffffff92279abbc0 : 0xffffff801145a734 
0xffffff92279abea0 : 0xffffff8011452097 
0xffffff92279abf20 : 0xffffff80118c1a7a 
0xffffff92279abf40 : 0xffffff80119b5aab 
0xffffff92279abfa0 : 0xffffff801135a466 

BSD process name corresponding to current thread: go-langserver

Mac OS version:
18B75

Kernel version:
Darwin Kernel Version 18.2.0: Fri Oct  5 19:41:49 PDT 2018; root:xnu-4903.221.2~2/RELEASE_X86_64
Kernel UUID: 5D53F7E4-472A-369D-97D8-4DD877A4BDFF
Kernel slide:     0x0000000011000000
Kernel text base: 0xffffff8011200000
__HIB  text base: 0xffffff8011100000
System model name: MacBookPro13,3 (Mac-A5C67F76ED83108C)

[sandipb]

Yeah, I can confirm. I have had one panic reboot when I let go-langserver run at top cpu load for too long.

[primalmotion]

I noticed that the golang servers that start to go crazy actually panic

It seems to panic, then it goes berserk on the CPU

panic serving textDocument/signatureHelp: illegal file offset
goroutine 48235 [running]:
github.com/sourcegraph/go-langserver/langserver/util.Panicf(0x1622320, 0x17f3a20, 0x17101ae, 0x2, 0xc1b2edf518, 0x1, 0x1, 0xc1b2edf528, 0x105b696)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/util/util.go:133 +0x93
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).Handle.func1(0xc0a5f1f540, 0xc1b2edfe60)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:164 +0xc4
panic(0x1622320, 0x17f3a20)
	/usr/local/Cellar/go/1.11.2/libexec/src/runtime/panic.go:513 +0x1b9
go/token.(*File).Pos(...)
	/usr/local/Cellar/go/1.11.2/libexec/src/go/token/position.go:252
github.com/sourcegraph/go-langserver/langserver.posForFileOffset(0xc0f82360c0, 0xc0725ed957, 0x82, 0x721, 0xc00026e080)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/loader.go:125 +0xd6
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).typecheck(0xc000274000, 0x17ffe80, 0xc01b4894d0, 0x17fdf80, 0xc00026e080, 0xc0725ed950, 0x89, 0x40, 0x5a, 0x0, ...)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/loader.go:81 +0x774
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handleTextDocumentSignatureHelp(0xc000274000, 0x17ffdc0, 0xc09201ea80, 0x17fdf80, 0xc00026e080, 0xc0a5f1f540, 0xc0725ed950, 0x89, 0x40, 0x5a, ...)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/signature.go:23 +0xcc
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).Handle(0xc000274000, 0x17ffdc0, 0xc09201ea80, 0x17fdf80, 0xc00026e080, 0xc0a5f1f540, 0x0, 0x0, 0x0, 0x0)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:417 +0x2c23
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handle(0xc000274000, 0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc0a5f1f540, 0xc000161350, 0xc022dbaf18, 0x13ceda9, 0xc00014d480)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:154 +0x66
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handle-fm(0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc0a5f1f540, 0x8, 0xc06008b440, 0x13c59ed, 0xc022dbaf30)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:30 +0x52
github.com/sourcegraph/go-langserver/vendor/github.com/sourcegraph/jsonrpc2.(*HandlerWithErrorConfigurer).Handle(0xc000070100, 0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc0a5f1f540)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/vendor/github.com/sourcegraph/jsonrpc2/handler_with_error.go:21 +0x73
created by github.com/sourcegraph/go-langserver/langserver.lspHandler.Handle
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:54 +0xfb
[Error - 12:37:38 PM] Request textDocument/signatureHelp failed.
  Message: unexpected panic: illegal file offset
  Code: 0 
panic serving textDocument/signatureHelp: illegal file offset
goroutine 48312 [running]:
github.com/sourcegraph/go-langserver/langserver/util.Panicf(0x1622320, 0x17f3a20, 0x17101ae, 0x2, 0xc0da60b518, 0x1, 0x1, 0xc0da60b528, 0x105b696)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/util/util.go:133 +0x93
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).Handle.func1(0xc1020496d0, 0xc0da60be60)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:164 +0xc4
panic(0x1622320, 0x17f3a20)
	/usr/local/Cellar/go/1.11.2/libexec/src/runtime/panic.go:513 +0x1b9
go/token.(*File).Pos(...)
	/usr/local/Cellar/go/1.11.2/libexec/src/go/token/position.go:252
github.com/sourcegraph/go-langserver/langserver.posForFileOffset(0xc101eb4b80, 0xc06f0cc6c7, 0x82, 0x722, 0xc00026e080)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/loader.go:125 +0xd6
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).typecheck(0xc000274000, 0x17ffe80, 0xc102135500, 0x17fdf80, 0xc00026e080, 0xc06f0cc6c0, 0x89, 0x40, 0x5b, 0x0, ...)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/loader.go:81 +0x774
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handleTextDocumentSignatureHelp(0xc000274000, 0x17ffdc0, 0xc101eb49c0, 0x17fdf80, 0xc00026e080, 0xc1020496d0, 0xc06f0cc6c0, 0x89, 0x40, 0x5b, ...)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/signature.go:23 +0xcc
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).Handle(0xc000274000, 0x17ffdc0, 0xc101eb49c0, 0x17fdf80, 0xc00026e080, 0xc1020496d0, 0x0, 0x0, 0x0, 0x0)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:417 +0x2c23
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handle(0xc000274000, 0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc1020496d0, 0xc000161350, 0xc001950718, 0x13ceda9, 0xc00014d480)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:154 +0x66
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handle-fm(0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc1020496d0, 0x8, 0xc00518ef40, 0x13c59ed, 0xc001950730)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:30 +0x52
github.com/sourcegraph/go-langserver/vendor/github.com/sourcegraph/jsonrpc2.(*HandlerWithErrorConfigurer).Handle(0xc000070100, 0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc1020496d0)
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/vendor/github.com/sourcegraph/jsonrpc2/handler_with_error.go:21 +0x73
created by github.com/sourcegraph/go-langserver/langserver.lspHandler.Handle
	/Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:54 +0xfb
[Error - 12:37:38 PM] Request textDocument/signatureHelp failed.
  Message: unexpected panic: illegal file offset
  Code: 0 

[primalmotion]

this diff should prevent the panic at least

diff --git a/langserver/loader.go b/langserver/loader.go
index f030f90..a0f50c5 100644
--- a/langserver/loader.go
+++ b/langserver/loader.go
@@ -122,6 +122,11 @@ func posForFileOffset(fset *token.FileSet, filename string, offset int) token.Po
        if f == nil {
                return token.NoPos
        }
+
+       if offset > f.Size() {
+               return token.NoPos
+       }
+
        return f.Pos(offset)
 }

[primalmotion]

While it certainly fixes the panic, go-langserver is still eating all cpu and memory...

[arowden]

I'm also having this issue on Mac High Sierra 10.13.6. Seems like a memory leak. My CPU usage is low, but memory usage climbs until the os crashes.

[primalmotion]

The only way to make it usable for me with vscode is to kill it every 60s in a background bash loop