go
go copied to clipboard
runtime: process hangs after moving to go 1.17
What version of Go are you using (go version
)?
$ go version 1.17.2
Does this issue reproduce with the latest release?
Yes with 1.17.6 Not tried with 1.18
What operating system and processor architecture are you using (go env
)?
Windows
go env
Output
$ go env
What did you do?
We migrated our go version from go1.12 to go 1.17.2. After that we see that our process hangs sometimes on windows os. Most of the times it runs fine, but some times it hangs and it never comes out. Even it doesn't return go profiles (we have pprof included in process). We get this issue in production, but not able to produce this in house, so dont have much details. I am attaching process dump that we have taken from explorer. I do see it doing force GC, but can't actually figure out what is the issue.
One more thing: We have this runtime.ReadMemStats() happening every 30 sec. Not sure if that is related, but just noting down here.
What did you expect to see?
Process should continue and finish gracefully.
What did you see instead?
Process hangs
dlv output from process dump
(dlv) goroutines
Goroutine 1 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.3206878s]
Goroutine 2 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [force gc (idle) 457422h31m33.7887784s]
Goroutine 3 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [finalizer wait 457422h31m16.0308288s]
Goroutine 4 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/vendor/github.com/golang/glog/glog.go:882 github.com/golang/glog.(*loggingT).flushDaemon (0x7ff69227e10a) [chan receive]
Goroutine 5 - User: c:/program files/go/src/runtime/syscall_windows.go:479 syscall.Syscall (0x7ff691f879f4) (thread 17308)
Goroutine 6 - User: c:/program files/go/src/net/fd_windows.go:139 net.(*netFD).accept (0x7ff69202c9c5) [IO wait 457422h31m33.8227753s]
Goroutine 10 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/httpframe.go:123 phoenix/pkg/druvarpc.(*HTTPReader).contLenValueState (0x7ff69226d8cc) [GC assist wait]
Goroutine 11 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/clientsend.go:104 phoenix/pkg/druvarpc.(*Client).sendWorker (0x7ff69226c4b1) [select]
Goroutine 18 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC sweep wait 457422h31m13.400698s]
Goroutine 19 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC scavenge wait 457422h31m13.4056892s]
Goroutine 21 - User: c:/program files/go/src/database/sql/sql.go:1196 database/sql.(*DB).connectionOpener (0x7ff692289c73) [select 457422h31m33.8647754s]
Goroutine 22 - User: c:/program files/go/src/database/sql/sql.go:1196 database/sql.(*DB).connectionOpener (0x7ff692289c73) [select 457422h31m33.873777s]
Goroutine 23 - User: c:/program files/go/src/database/sql/sql.go:1196 database/sql.(*DB).connectionOpener (0x7ff692289c73) [select 457422h31m33.8797796s]
Goroutine 24 - User: c:/program files/go/src/database/sql/sql.go:1196 database/sql.(*DB).connectionOpener (0x7ff692289c73) [select 457422h31m33.8887779s]
Goroutine 25 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/chunker.go:94 phoenix/pkg/cloudapi/bkppipe.chunkWorker (0x7ff692338b0d) [select 457422h31m13.4476949s]
Goroutine 26 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/chunker.go:94 phoenix/pkg/cloudapi/bkppipe.chunkWorker (0x7ff692338b0d) [select]
Goroutine 27 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/chunker.go:28 phoenix/pkg/cloudapi/bkppipe.chunkBuffer (0x7ff69233848c) [GC assist wait]
Goroutine 28 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/chunker.go:94 phoenix/pkg/cloudapi/bkppipe.chunkWorker (0x7ff692338b0d) [select]
Goroutine 29 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/hcache.go:71 phoenix/pkg/cloudapi/bkppipe.hcacheWorker (0x7ff69233b35c) [select 457422h31m13.4876916s]
Goroutine 30 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/hcache.go:71 phoenix/pkg/cloudapi/bkppipe.hcacheWorker (0x7ff69233b35c) [select]
Goroutine 31 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/hcache.go:71 phoenix/pkg/cloudapi/bkppipe.hcacheWorker (0x7ff69233b35c) [select]
Goroutine 32 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/hcache.go:71 phoenix/pkg/cloudapi/bkppipe.hcacheWorker (0x7ff69233b35c) [select 457422h31m13.5216927s]
Goroutine 33 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.5317277s]
Goroutine 34 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/vendor/gopkg.in/natefinch/lumberjack.v2/lumberjack.go:379 gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun (0x7ff69224cd05) [chan receive 457422h31m33.9938268s]
Goroutine 36 - User: c:/program files/go/src/runtime/mstats.go:403 runtime.ReadMemStats (0x7ff691f511ca) (thread 12680) [chan receive]
Goroutine 40 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/httpframe.go:123 phoenix/pkg/druvarpc.(*HTTPReader).contLenValueState (0x7ff69226d8cc) [GC assist wait]
Goroutine 41 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/clientsend.go:104 phoenix/pkg/druvarpc.(*Client).sendWorker (0x7ff69226c4b1) [select 457422h31m13.5717388s]
Goroutine 42 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/vendor/gopkg.in/natefinch/lumberjack.v2/lumberjack.go:379 gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun (0x7ff69224cd05) [chan receive 457422h31m34.03183s]
Goroutine 44 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/fileserver/client/agent/backupagent/backup_stats.go:147 phoenix/fileserver/client/agent/backupagent.(*statsCollector).start.func1 (0x7ff6923dc2cd) [chan receive 457422h31m16.280859s]
Goroutine 46 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/fileserver/client/upload/smap_worker.go:31 phoenix/fileserver/client/upload.(*Uploader).smapCommitWorker (0x7ff6923bf83f) [select 457422h31m13.6077414s]
Goroutine 50 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/client/autouploadlog/auto_log_upload.go:101 phoenix/pkg/client/autouploadlog.(*UploadLog).autoJobUploadTicker (0x7ff69231ee57) [chan receive 457422h31m34.0708297s]
Goroutine 54 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/httpframe.go:154 phoenix/pkg/druvarpc.(*HTTPReader).contentBufState (0x7ff69226daf0) [GC assist wait]
Goroutine 55 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/clientsend.go:104 phoenix/pkg/druvarpc.(*Client).sendWorker (0x7ff69226c4b1) [select]
Goroutine 56 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle) 457422h31m34.1108348s]
Goroutine 57 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle) 457422h31m34.1178326s]
Goroutine 58 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle) 457422h31m34.1238328s]
Goroutine 59 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle) 457422h31m34.132833s]
Goroutine 60 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 61 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 62 - User: :0 ??? (0x7ffbe7d75d04) (thread 17336) [GC worker (idle)]
Goroutine 63 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 64 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 65 - User: :0 ??? (0x7ffbe7d75d04) (thread 14176) [GC worker (idle)]
Goroutine 66 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.7257469s]
Goroutine 67 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.7367476s]
Goroutine 68 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.7467481s]
Goroutine 69 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.7557464s]
Goroutine 70 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.7657483s]
Goroutine 71 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.7767504s]
Goroutine 72 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select]
Goroutine 73 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select]
Goroutine 74 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/probe.go:100 phoenix/pkg/cloudapi/bkppipe.probeWorker (0x7ff692343e3c) [select 457422h31m13.8047517s]
Goroutine 75 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/probe.go:100 phoenix/pkg/cloudapi/bkppipe.probeWorker (0x7ff692343e3c) [select 457422h31m13.8157524s]
Goroutine 76 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/probe.go:100 phoenix/pkg/cloudapi/bkppipe.probeWorker (0x7ff692343e3c) [select 457422h31m13.8267532s]
Goroutine 77 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.8377535s]
Goroutine 78 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/probe.go:100 phoenix/pkg/cloudapi/bkppipe.probeWorker (0x7ff692343e3c) [select 457422h31m13.8517564s]
Goroutine 79 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/probe.go:100 phoenix/pkg/cloudapi/bkppipe.probeWorker (0x7ff692343e3c) [select 457422h31m13.8647577s]
Goroutine 80 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/probe.go:100 phoenix/pkg/cloudapi/bkppipe.probeWorker (0x7ff692343e3c) [select 457422h31m13.8777595s]
Goroutine 81 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/probe.go:100 phoenix/pkg/cloudapi/bkppipe.probeWorker (0x7ff692343e3c) [select 457422h31m13.8907533s]
Goroutine 82 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/probe.go:100 phoenix/pkg/cloudapi/bkppipe.probeWorker (0x7ff692343e3c) [select 457422h31m13.9007566s]
Goroutine 83 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/probe.go:100 phoenix/pkg/cloudapi/bkppipe.probeWorker (0x7ff692343e3c) [select 457422h31m13.9107547s]
Goroutine 84 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/probe.go:100 phoenix/pkg/cloudapi/bkppipe.probeWorker (0x7ff692343e3c) [select 457422h31m13.9237565s]
Goroutine 85 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.9357608s]
Goroutine 86 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 87 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.9527585s]
Goroutine 88 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.9617601s]
Goroutine 89 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m13.9707617s]
Goroutine 90 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/compression.go:72 phoenix/pkg/cloudapi/bkppipe.compressionWorker (0x7ff692339745) [select 457422h31m13.9817624s]
Goroutine 91 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/compression.go:72 phoenix/pkg/cloudapi/bkppipe.compressionWorker (0x7ff692339745) [select 457422h31m13.9917634s]
Goroutine 92 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/compression.go:72 phoenix/pkg/cloudapi/bkppipe.compressionWorker (0x7ff692339745) [select 457422h31m14.0037639s]
Goroutine 93 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.5927936s]
Goroutine 94 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m34.4788537s]
Goroutine 95 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.0417671s]
Goroutine 96 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.9266266s]
Goroutine 97 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.0657677s]
Goroutine 98 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.0777664s]
Goroutine 99 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.0877673s]
Goroutine 100 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.6787952s]
Goroutine 101 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.6907981s]
Goroutine 102 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.7018007s]
Goroutine 103 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.1347721s]
Goroutine 104 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.1457705s]
Goroutine 105 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.1637704s]
Goroutine 106 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.1737718s]
Goroutine 107 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m34.6358621s]
Goroutine 108 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.1977715s]
Goroutine 109 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m14.2087746s]
Goroutine 110 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m34.6688639s]
Goroutine 111 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m34.6848641s]
Goroutine 112 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m34.694866s]
Goroutine 113 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m34.7048678s]
Goroutine 114 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m34.7148669s]
Goroutine 115 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m34.724866s]
Goroutine 116 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m34.7368684s]
Goroutine 117 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/mput.go:284 phoenix/pkg/cloudapi/bkppipe.syncWorker (0x7ff69233dd1c) [select 457422h31m34.757871s]
Goroutine 118 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m34.7698711s]
Goroutine 119 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.6612659s]
Goroutine 120 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.340783s]
Goroutine 121 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.3517828s]
Goroutine 122 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.3647813s]
Goroutine 123 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.375782s]
Goroutine 124 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.7192667s]
Goroutine 125 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.7312696s]
Goroutine 126 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.7422671s]
Goroutine 127 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.7522703s]
Goroutine 130 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.4317851s]
Goroutine 131 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.4427896s]
Goroutine 132 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.4547883s]
Goroutine 133 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.4647892s]
Goroutine 134 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.4837978s]
Goroutine 135 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m14.4987898s]
Goroutine 136 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m34.9598789s]
Goroutine 137 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m34.9718823s]
Goroutine 138 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m34.9848803s]
Goroutine 139 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m34.9958834s]
Goroutine 140 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m35.0088838s]
Goroutine 141 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m35.0258862s]
Goroutine 142 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m35.0388921s]
Goroutine 143 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m35.0538856s]
Goroutine 144 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/postput.go:69 phoenix/pkg/cloudapi/bkppipe.postputWorker (0x7ff692342fbc) [select 457422h31m35.0648882s]
Goroutine 146 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle) 457422h31m35.0748905s]
Goroutine 147 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle) 457422h31m35.0838875s]
Goroutine 162 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle) 457422h31m35.0898905s]
Goroutine 163 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle) 457422h31m35.0958903s]
Goroutine 178 - User: c:/program files/go/src/runtime/mbitmap.go:514 runtime.scanobject (0x7ff691f402a9) (thread 17312) [GC worker (idle)]
Goroutine 179 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 180 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 181 - User: :0 ??? (0x7ffbe7d75d04) (thread 12956) [GC worker (idle)]
Goroutine 182 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 183 - User: :0 ??? (0x7ffbe7d75d04) (thread 16568) [GC worker (idle)]
Goroutine 184 - User: :0 ??? (0x7ffbe7d75d04) (thread 14092) [GC worker (idle)]
Goroutine 185 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 186 - User: :0 ??? (0x7ffbe7d75d04) (thread 17340) [GC worker (idle)]
Goroutine 187 - User: :0 ??? (0x7ffbe7d75d04) (thread 15324) [GC worker (idle)]
Goroutine 188 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 189 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 190 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 191 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 192 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 193 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 194 - User: :0 ??? (0x7ffbe7d76084) (thread 14372) [GC worker (idle)]
Goroutine 195 - User: c:/program files/go/src/runtime/proc.go:367 runtime.gopark (0x7ff691f5c796) [GC worker (idle)]
Goroutine 209 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m14.7388056s]
Goroutine 211 - User: c:/program files/go/src/database/sql/sql.go:1196 database/sql.(*DB).connectionOpener (0x7ff692289c73) [select 457422h31m35.197896s]
Goroutine 212 - User: c:/program files/go/src/database/sql/sql.go:1196 database/sql.(*DB).connectionOpener (0x7ff692289c73) [select 457422h31m35.2048951s]
Goroutine 226 - User: c:/program files/go/src/runtime/time.go:193 time.Sleep (0x7ff691f88213) [sleep 457422h31m35.2138953s]
Goroutine 227 - User: c:/program files/go/src/net/fd_windows.go:139 net.(*netFD).accept (0x7ff69202c9c5) [IO wait 457422h31m35.2188958s]
Goroutine 230 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/client/filenamelogger/filename_logger.go:95 phoenix/pkg/client/filenamelogger.(*FileNameLogger).fileLogWritter (0x7ff692360cdb) [select 457422h31m35.2248942s]
Goroutine 231 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/client/filenamelogger/filename_logger.go:95 phoenix/pkg/client/filenamelogger.(*FileNameLogger).fileLogWritter (0x7ff692360cdb) [select 457422h31m35.2448951s]
Goroutine 232 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/fileserver/client/upload/token.go:29 phoenix/fileserver/client/upload.(*Uploader).handleSmallFileUpload (0x7ff6923c5889) [chan receive 457422h31m14.8118087s]
Goroutine 233 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/fileserver/client/upload/uploader.go:134 phoenix/fileserver/client/upload.(*Uploader).deleteFileWorker (0x7ff6923c122f) [chan receive 457422h31m14.8248128s]
Goroutine 234 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/fileserver/client/upload/reader.go:63 phoenix/fileserver/client/upload.(*Uploader).fileOpener (0x7ff6923bf22d) [chan receive 457422h31m14.8378099s]
Goroutine 235 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/fileserver/client/upload/reader.go:63 phoenix/fileserver/client/upload.(*Uploader).fileOpener (0x7ff6923bf22d) [chan receive 457422h31m14.8498091s]
Goroutine 236 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/fileserver/client/upload/reader.go:49 phoenix/fileserver/client/upload.(*Uploader).fileReader (0x7ff6923bf097) [chan receive]
Goroutine 237 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/fileserver/client/upload/reader.go:49 phoenix/fileserver/client/upload.(*Uploader).fileReader (0x7ff6923bf097) [chan receive]
Goroutine 239 - User: c:/program files/go/src/runtime/sema.go:56 sync.runtime_Semacquire (0x7ff691f86385) [semacquire 457422h31m17.5689358s]
Goroutine 324 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/fileserver/client/upload/uploader.go:471 phoenix/fileserver/client/upload.(*Uploader).uploadFile (0x7ff6923c3f91) [GC assist wait]
Goroutine 326 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/fileserver/client/upload/uploader.go:471 phoenix/fileserver/client/upload.(*Uploader).uploadFile (0x7ff6923c3f91) [GC assist wait]
Goroutine 360 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m14.909817s]
Goroutine 557 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m14.9198235s]
Goroutine 558 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m14.9318268s]
Goroutine 559 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 560 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m14.9488162s]
Goroutine 561 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 660 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m14.9688162s]
Goroutine 661 - User: c:/program files/go/src/runtime/proc.go:372 sync.runtime_notifyListWait (0x7ff691f86686) [sync.Cond.Wait 457422h31m14.978819s]
Goroutine 662 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m14.9858209s]
Goroutine 663 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 664 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.0068212s]
Goroutine 665 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.0228229s]
Goroutine 666 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.0378219s]
Goroutine 667 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.0488222s]
Goroutine 668 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.0588245s]
Goroutine 669 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.0688241s]
Goroutine 670 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select]
Goroutine 671 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 672 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 673 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.1058237s]
Goroutine 674 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.1158265s]
Goroutine 675 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.1268305s]
Goroutine 676 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.1358274s]
Goroutine 690 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 691 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.1518286s]
Goroutine 692 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 693 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.1668295s]
Goroutine 694 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.1768295s]
Goroutine 695 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.1858297s]
Goroutine 696 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 697 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 698 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.2108306s]
Goroutine 699 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.2228405s]
Goroutine 700 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 701 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 702 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.2548341s]
Goroutine 703 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 704 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.279835s]
Goroutine 705 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 706 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 707 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 708 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.3118397s]
Goroutine 709 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 710 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.3278381s]
Goroutine 711 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 712 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.3438398s]
Goroutine 713 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.3528395s]
Goroutine 714 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.3618406s]
Goroutine 715 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 716 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 717 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.3848433s]
Goroutine 718 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 719 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 720 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.4108431s]
Goroutine 721 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 722 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 723 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.4338454s]
Goroutine 724 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.4438444s]
Goroutine 725 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.453843s]
Goroutine 726 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.4678469s]
Goroutine 727 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 728 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 729 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 730 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.499847s]
Goroutine 731 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 732 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.5168504s]
Goroutine 733 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 734 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.5358505s]
Goroutine 735 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.5478516s]
Goroutine 736 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 737 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 738 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.5778562s]
Goroutine 739 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.5888537s]
Goroutine 740 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 741 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 742 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.6178609s]
Goroutine 743 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 744 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.6338583s]
Goroutine 745 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 746 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.6488574s]
Goroutine 747 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 748 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.6698595s]
Goroutine 749 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.6818643s]
Goroutine 750 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.6918634s]
Goroutine 751 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 752 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.7078608s]
Goroutine 753 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.7178618s]
Goroutine 754 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.7278613s]
Goroutine 755 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.7368605s]
Goroutine 756 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.7468601s]
Goroutine 757 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.7568633s]
Goroutine 758 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.7698628s]
Goroutine 759 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.7808654s]
Goroutine 760 - User: c:/program files/go/src/runtime/sema.go:56 sync.runtime_Semacquire (0x7ff691f86385) [semacquire 457422h31m15.7908663s]
Goroutine 761 - User: c:/program files/go/src/runtime/sema.go:56 sync.runtime_Semacquire (0x7ff691f86385) [semacquire 457422h31m15.7968661s]
Goroutine 762 - User: c:/program files/go/src/runtime/sema.go:56 sync.runtime_Semacquire (0x7ff691f86385) [semacquire 457422h31m15.8038643s]
Goroutine 763 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/cloudapi/bkppipe/bufferpool.go:76 phoenix/pkg/cloudapi/bkppipe.getBuffer (0x7ff692337dfe) [GC assist wait]
Goroutine 764 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.8188671s]
Goroutine 765 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.8288685s]
Goroutine 766 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.8388666s]
Goroutine 767 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.8478654s]
Goroutine 768 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.8578673s]
Goroutine 769 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.8688708s]
Goroutine 770 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.8788718s]
Goroutine 771 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.8888708s]
Goroutine 772 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.9018745s]
Goroutine 773 - User: c:/program files/go/src/bytes/buffer.go:451 phoenix/pkg/druvarpc.(*Client).customDecoder (0x7ff69226b7d3) [GC assist wait]
Goroutine 774 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.9208751s]
Goroutine 775 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.931874s]
Goroutine 818 - User: c:/gitlab-runner/builds/hg6ddd65/0/phoenixdev/phoenix/go_workspace/src/phoenix/pkg/druvarpc/client.go:164 phoenix/pkg/druvarpc.(*Client).Call (0x7ff69226b5c5) [select 457422h31m15.9418735s]
[268 goroutines]
(dlv)
Is this golang issue or something else? We still have no clue why it happens randomly.
Does this issue also occur with Go 1.19?
A few things I notice of interest:
- Goroutine 5 is in syscall.Syscall. What syscall is this?
- Lots of goroutines are in "GC assist wait", yet GC doesn't really seem to be running (all workers are idle). @mknyszek does this seem abnormal to you?
- Goroutine 36 is in ReadMemStats, waiting for STW I think??
We are not able to reproduce this issue in house. It happens on our client machines where we dont have access. When we set "asyncpreemptoff=1", issue does not occur. For now we have set this variable and it is running fine with Go1.17.
One more thing, we have this code that periodically scans memory stats. Do you see any problem because of this?
func MonitorMemory(size uint64) {
var m runtime.MemStats
ticker := time.NewTicker(30 * time.Second)
defer ticker.Stop()
for range ticker.C {
runtime.ReadMemStats(&m)
if m.HeapIdle-m.HeapReleased > 1280*1024*1024 {
log.Debug("Heap released -->", m.HeapReleased, " Heap ideal -->", m.HeapIdle)
log.Debug("Freeing OS memory")
debug.FreeOSMemory()
freeOSMemoryCallCount++
}
}
}
I don't see anything fundamentally problematic with that loop, though you may consider if setting a memory limit (new in 1.19) is a better fit for your application: https://pkg.go.dev/runtime/debug#SetMemoryLimit
My best guess is it's stuck trying to STW because there's a bunch of goroutines stuck in semacquire (probably the STW sema for a GC). There's also a goroutine suspiciously in syscall_windows.go in the runtime.
If you're willing to share full goroutine stacks, that'll help a good bit in determining the root cause.
Actually no, taking a closer look, there are a lot of goroutines stuck in GC assist wait
and there is actually an idle mark worker in scanobject
. This seems like it's probably trying to STW for mark termination? Or is otherwise stuck in mark termination.
Thanks @prattmic, I will use that SetMemoryLimit call when we upgrade to Go 1.19.
@mknyszek I dont have full goroutine stack. We are not able to take goroutine trace via pprof when process hangs. We are not able to take any kind of traces via pprof. The attached one we obtained via windows process dump.
The attached one we obtained via windows process dump.
I see that you're checking the process dump in Delve; shouldn't that be able to provide a backtrace of each goroutine?
I believe I am having the exact same issue on Windows (x64), but I managed to get a stack trace of all goroutines (see attached files). From what I can see, every "active" goroutine seems to be waiting on GC and all goroutines involved in GC are "idle". While some other goroutines are waiting on channels/selects/IO, as soon as one attempts to allocate memory they hang as well. For example, I have enabled GoLang's built-in HTTP profiler. When the app hangs, I can fetch the "/debug/pprof/" home page exactly once, but if I attempt to click on any links to gather profiling info, that goroutine hangs indefinitely (and I can no longer even fetch the home page).
To obtain the attached stack trace, I had to make a build with debug symbols, give it to the customer, wait for it to hang, and then use "delve attach" to the hung PID and "goroutines -t"...
Note: I have seen this happen with builds made using GoLang versions 1.16.6 and 1.18.3. I have not tried 19.1 yet, or "asyncpreemptoff=1", and I also periodically call debug.FreeOSMemory() from main() (goroutine 1, which is not even a little close to the FreeOSMemory call when the app hangs). I do not use runtime.ReadMemStats the way sanjayvora does, I just call FreeOSMemory once every N minutes.
Update: I almost forgot. We have never seen this occur on any development system, and we have several customers running this application 24/7 who have also never encountered it. So far it has happened on one customer's system, but it occurs very often, and that customer happens to have McAfee Endpoint Security installed. McAfee doesn't log any complaints about our app (and we see nothing about it in Windows Event Manager). It may not be a factor, but it wouldn't be the first time I've seen anti-virus drivers do unexpected things to apps. We may ask the customer to temporarily remove McAfee to see if it makes a difference, but if something like "asyncpreemptoff=1" can fix it...
I believe that I stumbled across the same problem as described here. I wrote down my finding and thoughts in a containerd issue: https://github.com/containerd/containerd/issues/6362#issuecomment-1303175894 Maybe the findings are of help here and lead to an idea what's happening.
Update: Enabling GC traces revealed that right before the process gets unblocked, a forced GC is happening via the sysmon goroutine.
Any updates for this issue? It seems that my applicatin also encounter such problem in linux. we use go1.17.8, and here is the gc log of my application
gc 2195 @61793.198s 1%: 0.25+1357+614153 ms clock, 2.7+48/4051/182+6755686 ms cpu, 23833->24648->8332 MB, 25088 MB goal, 11 P
The GC log shows that the second STW lasts for about 10 minutes. But after that, the application can automatically return to normal.
@codablock Sorry, somehow I totally missed your comment from back in November. That sounds like a different issue (this seems pretty Windows-specific; ~there's Windows syscall callback functionality that doesn't exist on other platforms and it seems to be implicated here, re-reading the thread~ EDIT: Misread the above. I still think this is more likely Windows-specific).
Also, reading your linked comment, I'm not sure I follow the conclusion that it's a bug in the Go runtime. It sounds like the process is getting stuck, but I don't think the ways you tried to perturb it would really shift what the Go runtime is doing. My gut feeling is an OS issue, but I could be wrong.
@Jason7602 That also sounds like a different issue for the same reasons as above. In fact that's... quite surprising. Please file a new issue. Can you reproduce with the latest version of Go? Go 1.17 is no longer supported.
Actually, looking at the goroutine dump earlier on, this looks like an issue I've seen before. I think it was around Go 1.17 as well, and I think we fixed it. What's really suspicious is the extremely long wait times reported (52 years?). Let me try to dig up the other issue, this particular issue might be fixed.
Re-adding my 2 cents about the 1.17 question, which was:
Note: I have seen this happen with builds made using GoLang versions 1.16.6 and 1.18.3. I have not tried 19.1 yet
Problem: I can't reproduce this on any of my own development systems. I can only reproduce it on a customer's system, and I can't waste a customer's time hunting down an issue like this when I have a workaround. I believe the customer had anti-virus installed, and it's possible that some driver-level bug specific to that anti-virus was causing it, but I really have no idea. If I ever manage to reproduce this on a development system, I will add more info.
On Mon, Apr 10, 2023 at 12:00 PM Michael Knyszek @.***> wrote:
@codablock https://github.com/codablock Sorry, somehow I totally missed your comment from back in November. That sounds like a different issue (this seems pretty Windows-specific; there's Windows syscall callback functionality that doesn't exist on other platforms and it seems to be implicated here, re-reading the thread).
Also, reading your linked comment, I'm not sure I follow the conclusion that it's a bug in the Go runtime. It sounds like the process is getting stuck, but I don't think the ways you tried to perturb it would really shift what the Go runtime is doing. My gut feeling is an OS issue, but I could be wrong.
@Jason7602 https://github.com/Jason7602 That also sounds like a different issue for the same reasons as above. In fact that's... quite surprising. Please file a new issue. Can you reproduce with the latest version of Go? Go 1.17 is no longer supported.
— Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/52178#issuecomment-1501989688, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7JSCXP3X34EJHWFNUBA6TXAQVAVANCNFSM5SVOEIRQ . You are receiving this because you commented.Message ID: @.***>
@codablock Sorry, somehow I totally missed your comment from back in November. That sounds like a different issue (this seems pretty Windows-specific; ~there's Windows syscall callback functionality that doesn't exist on other platforms and it seems to be implicated here, re-reading the thread~ EDIT: Misread the above. I still think this is more likely Windows-specific).
Also, reading your linked comment, I'm not sure I follow the conclusion that it's a bug in the Go runtime. It sounds like the process is getting stuck, but I don't think the ways you tried to perturb it would really shift what the Go runtime is doing. My gut feeling is an OS issue, but I could be wrong.
@Jason7602 That also sounds like a different issue for the same reasons as above. In fact that's... quite surprising. Please file a new issue. Can you reproduce with the latest version of Go? Go 1.17 is no longer supported.
@mknyszek Thank you for your reply. This is really a surprising phenomenon. It may be an OS problem, as you speculated, because if this is a bug in the go runtime, it is unlikely that it can be recovered after it hangs. In fact, this problem is not easy to reproduce. I will reproduce this problem as soon as possible, and then file a new issue and check whether this problem exists in the latest Go version.
Actually, looking at the goroutine dump earlier on, this looks like an issue I've seen before. I think it was around Go 1.17 as well, and I think we fixed it. What's really suspicious is the extremely long wait times reported (52 years?). Let me try to dig up the other issue, this particular issue might be fixed.
Hi @mknyszek
Can we know which version of golang it was fixed?
What's really suspicious is the extremely long wait times reported (52 years?)
Delve bug, please disregard.
Actually, looking at the goroutine dump earlier on, this looks like an issue I've seen before. I think it was around Go 1.17 as well, and I think we fixed it. What's really suspicious is the extremely long wait times reported (52 years?). Let me try to dig up the other issue, this particular issue might be fixed.
Hi @mknyszek
Can we know which version of golang it was fixed?
Everyone seems to be ignoring my reply (which was merely to call out my post from 2022-09-21), but if he thinks it was fixed during the 1.17 release cycle... I was still able to reproduce it on GoLang versions 1.16.6 through 1.18.3, so I do not believe it was fixed during the 1.17 release cycle unless it was a late fix that was made after the 1.18.3 release.
IMHO, the biggest problem with trying to eliminate this problem is how difficult it is to reproduce. I only saw the problem on 1 customer system (out of hundreds that ran our GoLang apps with no problems). That 1 customer was able to reproduce it fairly reliably, but only after running our product 24/7 for several days, so each test cycle was very long. The only thing that stood out on his server was McAfee Endpoint Security. That may have had something to do with it, but it may not have. Disabling async preemption fixed the problem permanently, and as we never managed to reproduce the problem in-house and we cannot use customers as guinea pigs, we just chose to disable that GoLang option permanently forever in all of our Windows GoLang apps.
If it helps at all, my earlier post did include a full goroutine and environment dumps from the customer when the problem occurred.
Everyone seems to be ignoring my reply (which was merely to call out my post from 2022-09-21), but if he thinks it was fixed during the 1.17 release cycle... I was still able to reproduce it on GoLang versions 1.16.6 through 1.18.3, so I do not believe it was fixed during the 1.17 release cycle unless it was a late fix that was made after the 1.18.3 release.
Yeah, sorry. Missed that. And based on @aarzilli's comment, that long wait time was a red herring. I take back my speculation about this possibly being fixed.
Can we know which version of golang it was fixed?
See above, I don't think it was.
Disabling async preemption fixed the problem permanently, and as we never managed to reproduce the problem in-house and we cannot use customers as guinea pigs, we just chose to disable that GoLang option permanently forever in all of our Windows GoLang apps.
Got it. Async preemption does appear to be the common piece here. I dug up this open issue (#36492) that also gets resolved with asyncpreemptoff=1
on Windows machines (though that bug was about Windows 7 specifically?). It seems possibly related?
FTR we're running our full test suite against Windows machines (from different eras) regularly, and digging around our open issues (with the caveat that GitHub search is a bit hard to work with) I don't see any Windows-specific test timeout issues. I'm inclined to believe that what's happening is this feature is not playing well with some version of Windows, or something in the environment on some Windows machines. We've also fixed async preemption bugs over the years, so it's possible that now we've hidden the issue. :(
At this point we're almost certainly not going to go back and fix up Go 1.17 as per our policy, but at the very least I can file an issue to catch any Windows timeouts on our builders.
Filed #59576 to track any failures we see on our builders. It's closed right now on purpose; if we see any failures the bot will open the issue and we'll get it in the triage queue.
Disabling async preemption fixed the problem permanently, and as we never managed to reproduce the problem in-house and we cannot use customers as guinea pigs, we just chose to disable that GoLang option permanently forever in all of our Windows GoLang apps.
Got it. Async preemption does appear to be the common piece here. I dug up this open issue (#36492) that also gets resolved with
asyncpreemptoff=1
on Windows machines (though that bug was about Windows 7 specifically?). It seems possibly related?FTR we're running our full test suite against Windows machines (from different eras) regularly, and digging around our open issues (with the caveat that GitHub search is a bit hard to work with) I don't see any Windows-specific test timeout issues. I'm inclined to believe that what's happening is this feature is not playing well with some version of Windows, or something in the environment on some Windows machines. We've also fixed async preemption bugs over the years, so it's possible that now we've hidden the issue. :(
At this point we're almost certainly not going to go back and fix up Go 1.17 as per our policy, but at the very least I can file an issue to catch any Windows timeouts on our builders.
Agreed. I've seen it on recent versions of Windows Server (definitely way beyond Windows 7). While this may not help you reproduce it (especially as code changes over time), my recommendation would be:
- Set up a test run using the latest release/build with async preemption enabled.
- Test specific scenarios that actually need async preemption enabled (just in case that helps trigger it).
- Test it with CGo using gcc to keep it similar to my go environment (may not be needed at all, and I primarily use CGo for SQLite).
- During the test run, call debug.FreeOSMemory() often.
- Let the test run 24/7 for days if it doesn't lock up sooner.
- (Maybe) Install McAfee and see if it makes a difference.
(...) McAfee (...) asyncpreemption
See also #42528. I wonder if these anti viruses aren't doing debugger-like things and running into some variant of #36494.
Hi all, I don't know if this helps, but I had this exact same issue but finally tracked it down to a coding error. Before 1.17, the coding error never caused any issues. In Linux, it still didn't cause an issue but in Windows it sometimes happened with the process just hanging with lots of processes stuck. Could be reproduced on the same box running Windows & but not on partition running Linux in separate partitions. The scenario I have is a lot of threads analysing stuff, and using a key to read a struct from a map as part of the process. If the key is not in the map then initialise that key with a default struct. Because the map is read so much the routine to look up the struct favours read lock over write lock. My code is as follows (simplified):
var exists bool
mapMutex.RLock()
m, exists = map[k]
mapMutex.RUnlock()
if !exists {
mapMutex.Lock()
// Check someone didn't create whilst we waited to get the lock
if m, exists = map[k]; !exists {
m = &struct{....}
map[k] = m
}
mapMutex.Unlock()
}
The fault in the code was that I had the unlock after map[k] = m
. So if some other process had actually added to the map then the lock was never released. Basically, a very low probability scenario. As I say before 1.17 it never happened then it started to happen after 1.17, but only on Windows, but a resubmission of the request by the client then killed the stalled process and the next run was OK. I actually saw this issue being raised and thought someone would solve, then after 1.21 I decided to do some more analysis.
I don't know if this helps, but maybe the fault is closer to home than you think.