go icon indicating copy to clipboard operation
go copied to clipboard

cmd/go: go list is incredibly slow on Windows 11

Open wes-mil opened this issue 3 months ago • 14 comments

Go version

go version go1.25.0 windows/amd64

Output of go env in your module/workspace:

set AR=ar
set CC=gcc
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_ENABLED=0
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set CXX=g++
set GCCGO=gccgo
set GO111MODULE=
set GOAMD64=v1
set GOARCH=amd64
set GOAUTH=netrc
set GOBIN=
set GOCACHE=C:\Users\{user}\AppData\Local\go-build
set GOCACHEPROG=
set GODEBUG=
set GOENV=C:\Users\{user}\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFIPS140=off
set GOFLAGS=
set GOGCCFLAGS=-m64 -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=C:\Users\{user}\AppData\Local\Temp\go-build1520687272=/tmp/go-build -gno-record-gcc-switches
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMOD=C:\Users\{user}\source\testing\go.mod
set GOMODCACHE=C:\Users\{user}\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\{user}\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTELEMETRY=local
set GOTELEMETRYDIR=C:\Users\{user}\AppData\Roaming\go\telemetry
set GOTMPDIR=
set GOTOOLCHAIN=auto
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.25.0
set GOWORK=
set PKG_CONFIG=pkg-config

What did you do?

When running go list on Windows, I'm getting some absolutely unbearable performance. This is notably being run from gopls which I have created an issue for here #75208, however I've managed to narrow down the culprit to the go commands being run within gopls.

The isolation of this issue I have boiled down to removing a single import from this very simple Go program with a completed empty go.mod.

package main

import (
	"flag"
	"fmt"
)

var testflag string

func main() {
	flag.StringVar(&testflag, "", "", "")

	fmt.Println("hi")
}

In testing I have tried disabling antivirus, clearing modcache, fully reinstalling go, and disabling all other VS Code extensions. I can say relatively confidently that this is not a VS Code related issue.

What did you see happen?

In my example, removing that single import takes roughly 5-10 seconds to be reconciled by gopls.

I'm seeing log output like

3.2420477s for GOROOT= GOPATH=C:\Users\{user}\go GO111MODULE=auto GOPROXY=off PWD=C:\Users\{user}\source\testing go list ...
1.6115993s for GOROOT= GOPATH=C:\Users\{user}\go GO111MODULE=auto GOPROXY=off PWD=C:\Users\{user}\source\testing go list ...

The full log output is linked in the original issue #75208.

I then chased this issue down further referencing #71981.

Using command go list -e -json=... -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -debug-actiongraph="list.json" I get the follow output

  0.627s   7.06%  build sync/atomic
  0.626s  14.10%  build internal/cpu
  0.318s  17.69%  build internal/goexperiment
  0.317s  21.26%  build unicode
  0.317s  24.83%  build internal/goarch
  0.317s  28.40%  build unicode/utf8
  0.316s  31.96%  build unicode/utf16
  0.316s  35.52%  build structs
  0.316s  39.09%  build internal/goos
  0.316s  42.65%  build internal/itoa
  0.316s  46.21%  build math/bits
  0.316s  49.77%  build internal/trace/tracev2
  0.316s  53.34%  build internal/asan
  0.316s  56.90%  build internal/msan
  0.315s  60.45%  build internal/unsafeheader
  0.315s  64.00%  build internal/profilerecord
  0.315s  67.55%  build encoding
  0.315s  71.10%  build internal/coverage/rtcov
  0.315s  74.65%  build internal/syscall/windows/sysdll
  0.315s  78.20%  build internal/byteorder

And using go list -e -json=... -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -debug-actiongraph="list.json" -p=1 I get better performance but only marginally.

  0.311s  31.41%  build encoding
  0.308s  62.48%  link  test
  0.308s  93.52%  build internal/abi
  0.013s  94.83%  build runtime
  0.006s  95.39%  build math
  0.003s  95.65%  build internal/syscall/windows
  0.003s  95.90%  build os
  0.002s  96.11%  build internal/goexperiment
  0.002s  96.26%  build internal/syscall/execenv
  0.002s  96.41%  build unicode
  0.002s  96.56%  build iter
  0.002s  96.72%  build sync/atomic
  0.002s  96.87%  build internal/oserror
  0.001s  96.98%  build internal/runtime/sys
  0.001s  97.09%  build internal/syscall/windows/registry
  0.001s  97.20%  build reflect
  0.001s  97.31%  build test
  0.001s  97.41%  build internal/chacha8rand
  0.001s  97.52%  build internal/cpu
  0.001s  97.62%  build internal/race

Both of these, including the -p=1, are wildly slower than they should be. The performance I get on WSL2 locally and Pop OS! on a separate system is closer to 200ms.

What did you expect to see?

Realistically, a full reconciled code action should be taking 200-500ms in VS Code rather than 5-10s

wes-mil avatar Sep 02 '25 20:09 wes-mil

At this point you could try to bisect to find out which 1.25 commit introduced this, that would be really helpful.

ALTree avatar Sep 03 '25 07:09 ALTree

cc @golang/command-line

prattmic avatar Sep 03 '25 14:09 prattmic

I never actually ran go 1.24 on Windows. The only reason I used it over WSL2 is because I had a require that required Serial access and WSL2 passthrough wasn't cooperating. I have rerun these tests on go 1.24 and things have gotten substantially worse.

These tests are run with

go version go1.24.6 windows/amd64

Without -p=1

  8.693s   5.97%  build internal/cpu
  7.457s  11.09%  build internal/runtime/atomic
  6.839s  15.79%  build sync/atomic
  6.516s  20.26%  build internal/goexperiment
  6.515s  24.74%  build unicode
  6.515s  29.21%  build unicode/utf8
  6.515s  33.69%  build internal/goarch
  6.514s  38.16%  build internal/goos
  6.514s  42.63%  build math/bits
  6.514s  47.11%  build unicode/utf16
  6.514s  51.58%  build internal/asan
  6.514s  56.06%  build cmp
  6.514s  60.53%  build internal/coverage/rtcov
  6.514s  65.00%  build internal/syscall/windows/sysdll
  6.513s  69.48%  build internal/msan
  6.513s  73.95%  build internal/unsafeheader
  6.513s  78.42%  build internal/itoa
  6.513s  82.90%  build internal/byteorder
  6.513s  87.37%  build internal/godebugs
  6.513s  91.84%  build internal/profilerecord

With -p=1

  0.338s  31.96%  link  test
  0.338s  63.88%  build encoding
  0.325s  94.57%  build internal/abi
  0.014s  95.85%  build runtime
  0.005s  96.36%  build os
  0.005s  96.80%  build math
  0.003s  97.04%  build io/fs
  0.002s  97.24%  build internal/bytealg
  0.002s  97.43%  build internal/goexperiment
  0.002s  97.63%  build syscall
  0.002s  97.77%  build strconv
  0.002s  97.92%  build internal/coverage/rtcov
  0.001s  98.02%  build internal/race
  0.001s  98.12%  build unicode
  0.001s  98.22%  build reflect
  0.001s  98.32%  build slices
  0.001s  98.41%  build io
  0.001s  98.51%  build internal/reflectlite
  0.001s  98.61%  build sync
  0.001s  98.71%  build internal/runtime/maps

I will downgrade again and try again

wes-mil avatar Sep 03 '25 15:09 wes-mil

So maybe 1.25 is faster than 1.24 because it includes the fix for #71981? Have you tried uninstalling that "Microsoft PC Manager" mentioned in that issue? A few people there claimed it solved the slowness.

ALTree avatar Sep 03 '25 15:09 ALTree

The bizarre thing is that I don't have "Microsoft PC Manager" installed on my system at all. I tried to uninstall it

wes-mil avatar Sep 03 '25 15:09 wes-mil

Now with

go version go1.23.12 windows/amd64

Without -p=1

  9.276s   5.98%  build internal/runtime/atomic
  9.275s  11.95%  build sync/atomic
  7.660s  16.89%  build internal/cpu
  7.333s  21.61%  build internal/goos
  7.333s  26.33%  build unicode
  7.333s  31.06%  build internal/goarch
  7.333s  35.78%  build internal/msan
  7.333s  40.51%  build internal/profilerecord
  7.333s  45.23%  build internal/race
  7.333s  49.95%  build internal/godebugs
  7.333s  54.68%  build internal/coverage/rtcov
  7.333s  59.40%  build unicode/utf16
  7.333s  64.12%  build internal/byteorder
  7.333s  68.85%  build internal/unsafeheader
  7.333s  73.57%  build internal/asan
  7.333s  78.30%  build unicode/utf8
  7.332s  83.02%  build internal/itoa
  7.332s  87.74%  build encoding
  7.332s  92.47%  build internal/syscall/windows/sysdll
  7.332s  97.19%  build cmp

With -p=1

  0.337s  31.79%  link  test
  0.335s  63.47%  build encoding
  0.329s  94.50%  build internal/abi
  0.014s  95.80%  build runtime
  0.009s  96.61%  build fmt
  0.007s  97.28%  build time
  0.005s  97.77%  build reflect
  0.005s  98.20%  build math
  0.003s  98.49%  build io
  0.002s  98.67%  build internal/bytealg
  0.002s  98.81%  build strings
  0.001s  98.91%  build internal/runtime/atomic
  0.001s  99.01%  build internal/goexperiment
  0.001s  99.11%  build internal/goos
  0.001s  99.20%  build math/bits
  0.001s  99.26%  build errors
  0.001s  99.31%  build runtime/internal/math
  0.001s  99.36%  build internal/goarch
  0.001s  99.41%  build runtime/internal/sys
  0.001s  99.46%  build internal/unsafeheader

I don't think there's much value in continuing to downgrade versions, so I'm going to stop here. For the life of me, I can't seem to find a thread to pull here

wes-mil avatar Sep 03 '25 16:09 wes-mil

I've gone ahead and run go list -compiled with debug-runtime-time on both my host OS (windows) and WSL2. Here are the thread results.

Windows: 1200ms Image

WSL2 Ubuntu: 32ms Image

wes-mil avatar Sep 04 '25 01:09 wes-mil

show me you Control Panel @wes-mil 👀 At the bottom right corner

zBo1997 avatar Sep 08 '25 03:09 zBo1997

@zBo1997 I'm going to need you to be more specific

wes-mil avatar Sep 08 '25 03:09 wes-mil

~What's your Windows 11 version? We've noticed that Windows 11 since 23H2 has a performance regression regarding context switches. I don't know if that is related or not.~

linsite avatar Sep 09 '25 11:09 linsite

@linsite I'm on the latest stable build of Windows 11 24H2: 26100.4946

wes-mil avatar Sep 09 '25 14:09 wes-mil

@linsite I'm on the latest stable build of Windows 11 24H2: 26100.4946

Have you ever experienced the same slowness on Windows 10? If so, that would indicate a long-standing issue. ~Here’s what I observed on a Windows 10 VM—it indeed runs much slower than on an Ubuntu VM.~ I forgot to turn off the antivirus.

go list -e -compiled=true -test=true -export=false -deps=true -find=false -pgo=off --debug-trace='trace.out2.cache' testing/...

Image

linsite avatar Dec 01 '25 01:12 linsite

I don't believe I used Go on Windows prior to Windows 11. However, I don't recall having this serious of a performance issue a few years ago

wes-mil avatar Dec 01 '25 01:12 wes-mil