go runtime: fatal error: mcall called on m->g0 stack on Windows

Go version

go1.21.4

Output of `go env` in your module/workspace:

NOTE: We compile inside docker. The output below is from running go env within the docker container that we use. We cross-compile for linux, windows, and darwin. The workers having a problem in this case are running windows/amd64.

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2751320750=/tmp/go-build -gno-record-gcc-switches'

What did you do?

We have a cli program that runs on many thousands of VMs / bare metal workers as part of our larger CI setup. Recently, we've been seeing a non-trivial number of panics on a specific set of workers. These workers are bare metal AMD Threadripper machines running Windows 10. Below is some info from DXDiag. (I can get additional information if needed).

      Time of this report: 4/23/2024, 09:36:37
             Machine name: XXXXXXXX
               Machine Id: {F429D65A-427C-4F90-A587-8AB68B33FB57}
         Operating System: Windows 10 Enterprise 64-bit (10.0, Build 19045) (19041.vb_release.191206-1406)
                 Language: English (Regional Setting: English)
      System Manufacturer: Equus Computer Systems
             System Model: Nobilis
                     BIOS: F5c (type: UEFI)
                Processor: AMD Ryzen Threadripper 3990X 64-Core Processor  (64 CPUs), ~2.9GHz
                   Memory: 262144MB RAM
      Available OS Memory: 262032MB RAM
                Page File: 38200MB used, 668832MB available
              Windows Dir: C:\Windows
          DirectX Version: DirectX 12
      DX Setup Parameters: Not found
         User DPI Setting: 96 DPI (100 percent)
       System DPI Setting: 96 DPI (100 percent)
          DWM DPI Scaling: Disabled
                 Miracast: Available, no HDCP
Microsoft Graphics Hybrid: Not Supported
 DirectX Database Version: 1.0.8
           DxDiag Version: 10.00.19041.3636 64bit Unicode

We're seeing two types of panics:

fatal error: runtime: mcall called on m->g0 stack
init order seemingly not following the spec: https://go.dev/ref/spec#Package_initialization
- init() from a package is being called before the init() of a package it depends on
- Meaning a global variable is not yet populated
- Leading to a nil pointer dereference panic

I have included multiple call stacks of these panics below. NOTE: these callstacks all happen on different machines. But they only happen on the same "series" of machine, which is documented above.

All the panics seem to happen during the init() phase of the runtime start-up. The panics are not super reproduce-able. I'm only really seeing it due to our scale.

What did you see happen?

Below are the callstacks from a number of panics across multiple machines.

fatal error: runtime: mcall called on m->g0 stack
runtime stack:
runtime.throw({0x1801948?, 0x2f?})
	runtime/panic.go:1077 +0x65 fp=0xc00025bc20 sp=0xc00025bbf0 pc=0x60c7a5
runtime.badmcall(0xd0?)
	runtime/proc.go:503 +0x1f fp=0xc00025bc40 sp=0xc00025bc20 pc=0x60fa9f
runtime.badmcall(0x60db19)
	<autogenerated>:1 +0x25 fp=0xc00025bc58 sp=0xc00025bc40 pc=0x641ea5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00025bc70
stack: frame={sp:0xc00025bc40, fp:0xc00025bc58} stack=[0x6fc804000,0x6fc9ff7b0)
goroutine 1 [running, locked to thread]:
google.golang.org/genproto/googleapis/type/expr.init()
	google.golang.org/[email protected]/googleapis/type/expr/expr.pb.go:149 +0x69 fp=0xc00025be10 sp=0xc00025be08 pc=0xc11629
runtime.doInit1(0x243c990)
	runtime/proc.go:6740 +0xdd fp=0xc00025bf40 sp=0xc00025be10 pc=0x61bf1d
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc00025bfe0 sp=0xc00025bf40 pc=0x60f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc00025bfe8 sp=0xc00025bfe0 pc=0x63d241
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f5fa8 sp=0xc0000f5f88 pc=0x60f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f5fe0 sp=0xc0000f5fa8 pc=0x60f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x63d241
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7f78 sp=0xc0000f7f58 pc=0x60f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f7fc8 sp=0xc0000f7f78 pc=0x5f8ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f7fe0 sp=0xc0000f7fc8 pc=0x5edc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x63d241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 18 [GC scavenge wait]:
runtime.gopark(0xc0000440e0?, 0x1a65f48?, 0x1?, 0x0?, 0xc0002041a0?)
	runtime/proc.go:398 +0xce fp=0xc00020df70 sp=0xc00020df50 pc=0x60f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x2768440)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc00020dfa0 sp=0xc00020df70 pc=0x5f6369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc00020dfc8 sp=0xc00020dfa0 pc=0x5f68fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc00020dfe0 sp=0xc00020dfc8 pc=0x5edbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc00020dfe8 sp=0xc00020dfe0 pc=0x63d241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5
goroutine 19 [finalizer wait]:
runtime.gopark(0x400000?, 0x1000f9e70?, 0x0?, 0x0?, 0x27cba60?)
	runtime/proc.go:398 +0xce fp=0xc0000f9e28 sp=0xc0000f9e08 pc=0x60f40e
runtime.runfinq()
	runtime/mfinal.go:193 +0x107 fp=0xc0000f9fe0 sp=0xc0000f9e28 pc=0x5ecce7
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0x63d241
created by runtime.createfing in goroutine 1
	runtime/mfinal.go:163 +0x3d

fatal error: runtime: mcall called on m->g0 stack
runtime stack:
runtime.throw({0x1522440?, 0x20?})
	/usr/local/go/src/runtime/panic.go:1047 +0x65 fp=0xc00083f748 sp=0xc00083f718 pc=0x2bc045
runtime.badmcall(0xa?)
	/usr/local/go/src/runtime/proc.go:468 +0x25 fp=0xc00083f768 sp=0xc00083f748 pc=0x2bf2c5
runtime.badmcall(0x0)
	<autogenerated>:1 +0x2c fp=0xc00083f780 sp=0xc00083f768 pc=0x2ec86c
runtime.asyncPreempt2()
	/usr/local/go/src/runtime/preempt.go:308 +0x3f fp=0xc00083f7a0 sp=0xc00083f780 pc=0x2bd27f
runtime.asyncPreempt()
	/usr/local/go/src/runtime/preempt_amd64.s:50 +0xdb fp=0xc00083f928 sp=0xc00083f7a0 pc=0x2ebbbb
github.com/dustin/go-humanize.revfmap(0xc00024f680)
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:33 fp=0xc00083f930 sp=0xc00083f928 pc=0xe36ac0
github.com/dustin/go-humanize.init()
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:30 +0x142b fp=0xc00083f990 sp=0xc00083f930 pc=0xe3872b
runtime.doInit(0x2156de0)
	/usr/local/go/src/runtime/proc.go:6329 +0x12d fp=0xc00083fac0 sp=0xc00083f990 pc=0x2cb2ed
runtime.doInit(0x2157c80)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fbf0 sp=0xc00083fac0 pc=0x2cb231
runtime.doInit(0x215a020)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fd20 sp=0xc00083fbf0 pc=0x2cb231
runtime.doInit(0x2164660)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fe50 sp=0xc00083fd20 pc=0x2cb231
runtime.doInit(0x2151b00)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083ff80 sp=0xc00083fe50 pc=0x2cb231
runtime.main()
	/usr/local/go/src/runtime/proc.go:233 +0x1bf fp=0xc00083ffe0 sp=0xc00083ff80 pc=0x2be73f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00083ffe8 sp=0xc00083ffe0 pc=0x2ea561
goroutine 1 [runnable, locked to thread]:
github.com/dustin/go-humanize.revfmap(0xc00024f680?)
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:33 +0xf0 fp=0xc00083f930 sp=0xc00083f928 pc=0xe36bb0
github.com/dustin/go-humanize.init()
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:30 +0x142b fp=0xc00083f990 sp=0xc00083f930 pc=0xe3872b
runtime.doInit(0x2156de0)
	/usr/local/go/src/runtime/proc.go:6329 +0x12d fp=0xc00083fac0 sp=0xc00083f990 pc=0x2cb2ed
runtime.doInit(0x2157c80)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fbf0 sp=0xc00083fac0 pc=0x2cb231
runtime.doInit(0x215a020)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fd20 sp=0xc00083fbf0 pc=0x2cb231
runtime.doInit(0x2164660)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fe50 sp=0xc00083fd20 pc=0x2cb231
runtime.doInit(0x2151b00)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083ff80 sp=0xc00083fe50 pc=0x2cb231
runtime.main()
	/usr/local/go/src/runtime/proc.go:233 +0x1bf fp=0xc00083ffe0 sp=0xc00083ff80 pc=0x2be73f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00083ffe8 sp=0xc00083ffe0 pc=0x2ea561
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f3fb0 sp=0xc0000f3f90 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:302 +0xb1 fp=0xc0000f3fe0 sp=0xc0000f3fb0 pc=0x2be9b1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f3fe8 sp=0xc0000f3fe0 pc=0x2ea561
created by runtime.init.6
	/usr/local/go/src/runtime/proc.go:290 +0x25
goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010ff90 sp=0xc00010ff70 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00010ffc8 sp=0xc00010ff90 pc=0x2a7c37
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x29c786
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x2ea561
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:178 +0x6b
goroutine 19 [GC scavenge wait]:
runtime.gopark(0xc000106000?, 0x179b290?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000111f70 sp=0xc000111f50 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x2432600)
	/usr/local/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc000111fa0 sp=0xc000111f70 pc=0x2a5c73
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc000111fc8 sp=0xc000111fa0 pc=0x2a6285
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000111fe0 sp=0xc000111fc8 pc=0x29c726
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x2ea561
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:179 +0xaa
goroutine 34 [finalizer wait]:
runtime.gopark(0x0?, 0xc0000f7e70?, 0x6b?, 0x7d?, 0xc0000f7f70?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f7e28 sp=0xc0000f7e08 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0xc0000f7fe0 sp=0xc0000f7e28 pc=0x29b88f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x2ea561
created by runtime.createfing
	/usr/local/go/src/runtime/mfinal.go:157 +0x45
goroutine 35 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010bf50 sp=0xc00010bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00010bfe0 sp=0xc00010bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 3 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f5f50 sp=0xc0000f5f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000f5fe0 sp=0xc0000f5f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 20 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011df50 sp=0xc00011df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011dfe0 sp=0xc00011df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 21 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011ff50 sp=0xc00011ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011ffe0 sp=0xc00011ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011ffe8 sp=0xc00011ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 36 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010df50 sp=0xc00010df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00010dfe0 sp=0xc00010df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 37 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000119f50 sp=0xc000119f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000119fe0 sp=0xc000119f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 4 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000505f50 sp=0xc000505f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000505fe0 sp=0xc000505f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000505fe8 sp=0xc000505fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 5 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000507f50 sp=0xc000507f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000507fe0 sp=0xc000507f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 6 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000501f50 sp=0xc000501f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000501fe0 sp=0xc000501f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000501fe8 sp=0xc000501fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 22 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000127f50 sp=0xc000127f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000127fe0 sp=0xc000127f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000127fe8 sp=0xc000127fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 23 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000129f50 sp=0xc000129f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000129fe0 sp=0xc000129f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000129fe8 sp=0xc000129fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 38 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011bf50 sp=0xc00011bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011bfe0 sp=0xc00011bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 7 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000503f50 sp=0xc000503f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000503fe0 sp=0xc000503f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000503fe8 sp=0xc000503fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 24 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000123f50 sp=0xc000123f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000123fe0 sp=0xc000123f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000123fe8 sp=0xc000123fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 25 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000125f50 sp=0xc000125f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000125fe0 sp=0xc000125f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000125fe8 sp=0xc000125fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 8 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050df50 sp=0xc00050df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050dfe0 sp=0xc00050df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 26 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000509f50 sp=0xc000509f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000509fe0 sp=0xc000509f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000509fe8 sp=0xc000509fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 9 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050ff50 sp=0xc00050ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050ffe0 sp=0xc00050ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050ffe8 sp=0xc00050ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 27 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050bf50 sp=0xc00050bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050bfe0 sp=0xc00050bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050bfe8 sp=0xc00050bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 10 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000515f50 sp=0xc000515f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000515fe0 sp=0xc000515f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000515fe8 sp=0xc000515fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 28 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000511f50 sp=0xc000511f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000511fe0 sp=0xc000511f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000511fe8 sp=0xc000511fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 11 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000517f50 sp=0xc000517f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000517fe0 sp=0xc000517f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000517fe8 sp=0xc000517fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 29 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000513f50 sp=0xc000513f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000513fe0 sp=0xc000513f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000513fe8 sp=0xc000513fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 12 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051df50 sp=0xc00051df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051dfe0 sp=0xc00051df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051dfe8 sp=0xc00051dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 30 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000519f50 sp=0xc000519f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000519fe0 sp=0xc000519f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000519fe8 sp=0xc000519fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 13 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051ff50 sp=0xc00051ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051ffe0 sp=0xc00051ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051ffe8 sp=0xc00051ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 31 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051bf50 sp=0xc00051bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051bfe0 sp=0xc00051bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051bfe8 sp=0xc00051bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 14 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000525f50 sp=0xc000525f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000525fe0 sp=0xc000525f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000525fe8 sp=0xc000525fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 39 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000521f50 sp=0xc000521f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000521fe0 sp=0xc000521f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000521fe8 sp=0xc000521fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 15 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000527f50 sp=0xc000527f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000527fe0 sp=0xc000527f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000527fe8 sp=0xc000527fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 40 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000523f50 sp=0xc000523f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000523fe0 sp=0xc000523f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000523fe8 sp=0xc000523fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 16 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052ff50 sp=0xc00052ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052ffe0 sp=0xc00052ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052ffe8 sp=0xc00052ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 41 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052bf50 sp=0xc00052bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052bfe0 sp=0xc00052bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052bfe8 sp=0xc00052bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 50 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000531f50 sp=0xc000531f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000531fe0 sp=0xc000531f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000531fe8 sp=0xc000531fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 42 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052df50 sp=0xc00052df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052dfe0 sp=0xc00052df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052dfe8 sp=0xc00052dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 51 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000537f50 sp=0xc000537f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000537fe0 sp=0xc000537f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000537fe8 sp=0xc000537fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 43 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000533f50 sp=0xc000533f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000533fe0 sp=0xc000533f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000533fe8 sp=0xc000533fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 52 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000539f50 sp=0xc000539f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000539fe0 sp=0xc000539f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000539fe8 sp=0xc000539fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 44 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000535f50 sp=0xc000535f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000535fe0 sp=0xc000535f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000535fe8 sp=0xc000535fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 53 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053ff50 sp=0xc00053ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053ffe0 sp=0xc00053ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 45 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053bf50 sp=0xc00053bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053bfe0 sp=0xc00053bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053bfe8 sp=0xc00053bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 54 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000541f50 sp=0xc000541f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000541fe0 sp=0xc000541f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000541fe8 sp=0xc000541fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 46 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053df50 sp=0xc00053df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053dfe0 sp=0xc00053df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053dfe8 sp=0xc00053dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 55 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000547f50 sp=0xc000547f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000547fe0 sp=0xc000547f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000547fe8 sp=0xc000547fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 47 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000543f50 sp=0xc000543f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000543fe0 sp=0xc000543f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000543fe8 sp=0xc000543fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 56 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000549f50 sp=0xc000549f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000549fe0 sp=0xc000549f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000549fe8 sp=0xc000549fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 48 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000545f50 sp=0xc000545f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000545fe0 sp=0xc000545f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000545fe8 sp=0xc000545fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 57 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054ff50 sp=0xc00054ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054ffe0 sp=0xc00054ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054ffe8 sp=0xc00054ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 49 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054bf50 sp=0xc00054bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054bfe0 sp=0xc00054bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054bfe8 sp=0xc00054bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 58 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000551f50 sp=0xc000551f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000551fe0 sp=0xc000551f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000551fe8 sp=0xc000551fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 66 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054df50 sp=0xc00054df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054dfe0 sp=0xc00054df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054dfe8 sp=0xc00054dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 59 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000557f50 sp=0xc000557f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000557fe0 sp=0xc000557f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000557fe8 sp=0xc000557fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 67 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000553f50 sp=0xc000553f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000553fe0 sp=0xc000553f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000553fe8 sp=0xc000553fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 60 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000559f50 sp=0xc000559f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000559fe0 sp=0xc000559f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000559fe8 sp=0xc000559fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 68 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000555f50 sp=0xc000555f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000555fe0 sp=0xc000555f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000555fe8 sp=0xc000555fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 61 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055ff50 sp=0xc00055ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055ffe0 sp=0xc00055ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055ffe8 sp=0xc00055ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 69 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055bf50 sp=0xc00055bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055bfe0 sp=0xc00055bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055bfe8 sp=0xc00055bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 62 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000561f50 sp=0xc000561f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000561fe0 sp=0xc000561f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000561fe8 sp=0xc000561fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 70 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055df50 sp=0xc00055df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055dfe0 sp=0xc00055df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055dfe8 sp=0xc00055dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 63 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000567f50 sp=0xc000567f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000567fe0 sp=0xc000567f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000567fe8 sp=0xc000567fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 71 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000563f50 sp=0xc000563f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000563fe0 sp=0xc000563f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000563fe8 sp=0xc000563fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 64 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000569f50 sp=0xc000569f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000569fe0 sp=0xc000569f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000569fe8 sp=0xc000569fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 72 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000565f50 sp=0xc000565f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000565fe0 sp=0xc000565f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000565fe8 sp=0xc000565fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 65 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00056ff50 sp=0xc00056ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00056ffe0 sp=0xc00056ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00056ffe8 sp=0xc00056ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 83 [select]:
runtime.gopark(0xc000571f88?, 0x3?, 0xc8?, 0xc?, 0xc000571f72?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000571df8 sp=0xc000571dd8 pc=0x2beb16
runtime.selectgo(0xc000571f88, 0xc000571f6c, 0xc0001d2500?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000571f38 sp=0xc000571df8 pc=0x2ccebc
go.opencensus.io/stats/view.(*worker).start(0xc0001d2500)
	/builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:292 +0xad fp=0xc000571fc8 sp=0xc000571f38 pc=0x977c2d
go.opencensus.io/stats/view.init.0.func1()
	/builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:34 +0x26 fp=0xc000571fe0 sp=0xc000571fc8 pc=0x976e66
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000571fe8 sp=0xc000571fe0 pc=0x2ea561
created by go.opencensus.io/stats/view.init.0
	/builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:34 +0x8d

Then here are some callstacks from init's that are seemingly not following the right init order:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x5 pc=0xb84adc]
goroutine 1 [running]:
golang.org/x/text/internal/language.addTags({0x139, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
	golang.org/x/[email protected]/internal/language/match.go:99 +0x27c
golang.org/x/text/internal/language.Tag.Maximize(...)
	golang.org/x/[email protected]/internal/language/match.go:68
golang.org/x/text/language.init.0()
	golang.org/x/[email protected]/language/match.go:727 +0x26d

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x9 pc=0x132a0c0]
goroutine 1 [running]:
google.golang.org/protobuf/internal/filedesc.(*Enums).Len(0x3023620?)
	/builds/my-project/vendor/google.golang.org/protobuf/internal/filedesc/desc_list_gen.go:25
google.golang.org/protobuf/reflect/protoregistry.rangeTopLevelDescriptors({0x23b2710, 0xc0003fc000}, 0xc00035f288)
	/builds/my-project/vendor/google.golang.org/protobuf/reflect/protoregistry/registry.go:408 +0x5d
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc0000081f8, {0x23b2710?, 0xc0003fc000?})
	/builds/my-project/vendor/google.golang.org/protobuf/reflect/protoregistry/registry.go:149 +0x739
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0x1dafbd0, 0x2e}, {0x2d53d20, 0x1bf, 0x1bf}, 0x1, 0x0, 0x0, 0x0, {0x239d908, ...}, ...})
	/builds/my-project/vendor/google.golang.org/protobuf/internal/filedesc/build.go:112 +0x1d6
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x1dafbd0, 0x2e}, {0x2d53d20, 0x1bf, 0x1bf}, 0x1, 0x0, 0x0, 0x0, {0x0, ...}, ...}, ...})
	/builds/my-project/vendor/google.golang.org/protobuf/internal/filetype/build.go:138 +0x1b8
google.golang.org/genproto/googleapis/rpc/code.file_google_rpc_code_proto_init()
	/builds/my-project/vendor/google.golang.org/genproto/googleapis/rpc/code/code.pb.go:331 +0x198
google.golang.org/genproto/googleapis/rpc/code.init.0()
	/builds/my-project/vendor/google.golang.org/genproto/googleapis/rpc/code/code.pb.go:313 +0x17

What did you expect to see?

The program finishes the init() phase without panics

Apr 29 '24 16:04 RichieSams

Below is my go.mod file (with a few redactions)

module my-project/v3

require (
	github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2
	github.com/antlr4-go/antlr/v4 v4.13.0
	github.com/bmatcuk/doublestar v1.3.4
	github.com/bsipos/thist v1.0.0
	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
	github.com/dustin/go-humanize v1.0.1
	github.com/getsentry/sentry-go v0.27.0
	github.com/go-ole/go-ole v1.3.0
	github.com/gobwas/glob v0.2.3
	github.com/gofrs/flock v0.8.1
	github.com/gonum/stat v0.0.0-20181125101827-41a0da705a5b
	github.com/gorilla/handlers v1.5.2
	github.com/gorilla/mux v1.8.1
	github.com/hashicorp/go-multierror v1.1.1
	github.com/jotfs/fastcdc-go v0.2.0
	github.com/json-iterator/go v1.1.12
	github.com/juju/ratelimit v1.0.2
	github.com/karrick/godirwalk v1.16.2
	github.com/olekukonko/tablewriter v0.0.5
	github.com/pkg/errors v0.9.1
	github.com/restic/chunker v0.4.0
	github.com/satori/go.uuid v1.2.0
	github.com/schollz/progressbar/v3 v3.14.2
	github.com/sirupsen/logrus v1.9.3
	github.com/skratchdot/open-golang v0.0.0-20190104022628-a2dfa6d0dab6
	github.com/spf13/cobra v1.8.0
	github.com/spf13/viper v1.18.2
	github.com/stretchr/testify v1.9.0
	github.com/thediveo/enumflag v0.10.1
	github.com/tinylib/msgp v1.1.9
	github.com/xhit/go-str2duration/v2 v2.1.0
	internal-a v1.7.0
	internal-b v1.2.0
	internal-c v1.7.0
	internal-d/v2 v2.70.0
	go.opentelemetry.io/otel v1.25.0
	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.25.0
	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.25.0
	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.25.0
	go.opentelemetry.io/otel/metric v1.25.0
	go.opentelemetry.io/otel/sdk v1.25.0
	go.opentelemetry.io/otel/sdk/metric v1.25.0
	go.opentelemetry.io/otel/trace v1.25.0
	golang.org/x/sync v0.7.0
	golang.org/x/sys v0.19.0
	golang.org/x/term v0.19.0
	golang.org/x/text v0.14.0
	google.golang.org/grpc v1.63.2
	gopkg.in/yaml.v2 v2.4.0
)

require (
	cloud.google.com/go v0.112.0 // indirect
	cloud.google.com/go/compute v1.24.0 // indirect
	cloud.google.com/go/compute/metadata v0.2.3 // indirect
	cloud.google.com/go/iam v1.1.6 // indirect
	cloud.google.com/go/storage v1.36.0 // indirect
	github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 // indirect
	github.com/Microsoft/go-winio v0.6.0 // indirect
	github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
	github.com/Shopify/toxiproxy/v2 v2.4.0 // indirect
	github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af // indirect
	github.com/aws/aws-sdk-go v1.28.9 // indirect
	github.com/beorn7/perks v1.0.1 // indirect
	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
	github.com/cespare/xxhash/v2 v2.2.0 // indirect
	github.com/containerd/continuity v0.3.0 // indirect
	github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
	github.com/docker/cli v20.10.17+incompatible // indirect
	github.com/docker/docker v20.10.7+incompatible // indirect
	github.com/docker/go-connections v0.4.0 // indirect
	github.com/docker/go-units v0.4.0 // indirect
	github.com/fatih/color v1.14.1 // indirect
	github.com/felixge/httpsnoop v1.0.4 // indirect
	github.com/fogleman/gg v1.2.1-0.20190220221249-0403632d5b90 // indirect
	github.com/fsnotify/fsnotify v1.7.0 // indirect
	github.com/go-logr/logr v1.4.1 // indirect
	github.com/go-logr/stdr v1.2.2 // indirect
	github.com/gogo/protobuf v1.3.2 // indirect
	github.com/golang-jwt/jwt/v5 v5.0.0 // indirect
	github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.4 // indirect
	github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac // indirect
	github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82 // indirect
	github.com/gonum/integrate v0.0.0-20181209220457-a422b5c0fdf2 // indirect
	github.com/gonum/internal v0.0.0-20181124074243-f884aa714029 // indirect
	github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9 // indirect
	github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9 // indirect
	github.com/google/s2a-go v0.1.7 // indirect
	github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
	github.com/google/uuid v1.6.0 // indirect
	github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
	github.com/googleapis/gax-go/v2 v2.12.0 // indirect
	github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0 // indirect
	github.com/hashicorp/errwrap v1.0.0 // indirect
	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
	github.com/hashicorp/go-hclog v1.5.0 // indirect
	github.com/hashicorp/go-retryablehttp v0.7.0 // indirect
	github.com/hashicorp/hcl v1.0.0 // indirect
	github.com/imdario/mergo v0.3.12 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af // indirect
	github.com/jung-kurt/gofpdf v1.0.3-0.20190309125859-24315acbbda5 // indirect
	github.com/kylelemons/godebug v1.1.0 // indirect
	github.com/magiconair/properties v1.8.7 // indirect
	github.com/mattn/go-colorable v0.1.13 // indirect
	github.com/mattn/go-isatty v0.0.20 // indirect
	github.com/mattn/go-runewidth v0.0.14 // indirect
	github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
	github.com/minio/minio-go/v6 v6.0.46 // indirect
	github.com/minio/sha256-simd v0.1.1 // indirect
	github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
	github.com/mitchellh/go-homedir v1.1.0 // indirect
	github.com/mitchellh/mapstructure v1.5.0 // indirect
	github.com/moby/term v0.0.0-20201216013528-df9cb8a40635 // indirect
	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
	github.com/modern-go/reflect2 v1.0.2 // indirect
	github.com/opencontainers/go-digest v1.0.0 // indirect
	github.com/opencontainers/image-spec v1.0.2 // indirect
	github.com/opencontainers/runc v1.1.5 // indirect
	github.com/ory/dockertest/v3 v3.10.0 // indirect
	github.com/pelletier/go-toml/v2 v2.1.0 // indirect
	github.com/philhofer/fwd v1.1.2 // indirect
	github.com/pierrec/lz4 v2.0.5+incompatible // indirect
	github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
	github.com/prometheus/client_golang v1.12.1 // indirect
	github.com/prometheus/client_model v0.2.0 // indirect
	github.com/prometheus/common v0.32.1 // indirect
	github.com/prometheus/procfs v0.7.3 // indirect
	github.com/rivo/uniseg v0.4.7 // indirect
	github.com/russross/blackfriday/v2 v2.1.0 // indirect
	github.com/sagikazarmark/locafero v0.4.0 // indirect
	github.com/sagikazarmark/slog-shim v0.1.0 // indirect
	github.com/sourcegraph/conc v0.3.0 // indirect
	github.com/spf13/afero v1.11.0 // indirect
	github.com/spf13/cast v1.6.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/subosito/gotenv v1.6.0 // indirect
	github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
	github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
	github.com/xeipuuv/gojsonschema v1.2.0 // indirect
	go.opencensus.io v0.24.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.30.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 // indirect
	go.opentelemetry.io/proto/otlp v1.1.0 // indirect
	go.uber.org/atomic v1.9.0 // indirect
	go.uber.org/multierr v1.9.0 // indirect
	golang.org/x/crypto v0.21.0 // indirect
	golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
	golang.org/x/image v0.0.0-20190802002840-cff245a6509b // indirect
	golang.org/x/mod v0.13.0 // indirect
	golang.org/x/net v0.23.0 // indirect
	golang.org/x/oauth2 v0.17.0 // indirect
	golang.org/x/time v0.5.0 // indirect
	golang.org/x/tools v0.14.0 // indirect
	gonum.org/v1/plot v0.0.0-20190515093506-e2840ee46a6b // indirect
	google.golang.org/api v0.162.0 // indirect
	google.golang.org/appengine v1.6.8 // indirect
	google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de // indirect
	google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20240401170217-c3f982113cda // indirect
	google.golang.org/protobuf v1.33.0 // indirect
	gopkg.in/ini.v1 v1.67.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	nhooyr.io/websocket v1.8.10 // indirect
)

replace github.com/antlr/antlr4 v0.0.0-20180905200841-432022fc1ca0 => github.com/michilu/antlr4 v0.0.0-20180803091604-411960b1878f

replace github.com/uber/jaeger-lib v2.0.0+incompatible => github.com/uber/jaeger-lib v1.5.0

replace github.com/skratchdot/open-golang => github.com/seanhoughton/open-golang v0.0.0-20190322203133-7972e18a724d

replace git.apache.org/thrift.git => github.com/apache/thrift v0.12.0

go 1.21

Apr 29 '24 16:04 RichieSams

This seems to be related to: https://github.com/golang/go/issues/56774

But in that issue, the poster said updating to 1.21.3 fixed it. However, we're using 1.21.4, and still seeing the issue.

Apr 29 '24 16:04 RichieSams

Slightly different panic:

unexpected fault address 0x411a89e
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0x411a89e pc=0xe003e1]
goroutine 1 [running, locked to thread]:
runtime.throw({0x1f23e6f?, 0x0?})
	runtime/panic.go:1077 +0x65 fp=0xc0000fbd58 sp=0xc0000fbd28 pc=0xd9c7a5
runtime.sigpanic()
	runtime/signal_windows.go:377 +0xd0 fp=0xc0000fbda0 sp=0xc0000fbd58 pc=0xdb06b0
time.map.init.1()
	time/zoneinfo_abbrs_windows.go:154 +0x81 fp=0xc0000fbe00 sp=0xc0000fbda0 pc=0xe003e1
time.init()
	time/zoneinfo_abbrs_windows.go:15 +0x25 fp=0xc0000fbe10 sp=0xc0000fbe00 pc=0xe001a5
runtime.doInit1(0x2b81dc0)
	runtime/proc.go:6740 +0xdd fp=0xc0000fbf40 sp=0xc0000fbe10 pc=0xdabf1d
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc0000fbfe0 sp=0xc0000fbf40 pc=0xd9f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0xdcd141
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7fa8 sp=0xc0000f7f88 pc=0xd9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f7fe0 sp=0xc0000f7fa8 pc=0xd9f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0xdcd141
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f9f78 sp=0xc0000f9f58 pc=0xd9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f9fc8 sp=0xc0000f9f78 pc=0xd88ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f9fe0 sp=0xc0000f9fc8 pc=0xd7dc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0xdcd141
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x21b8428?, 0x1?, 0x0?, 0xc0000f4b60?)
	runtime/proc.go:398 +0xce fp=0xc000109f70 sp=0xc000109f50 pc=0xd9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x2e63e20)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc000109fa0 sp=0xc000109f70 pc=0xd86369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc000109fc8 sp=0xc000109fa0 pc=0xd868fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc000109fe0 sp=0xc000109fc8 pc=0xd7dbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc000109fe8 sp=0xc000109fe0 pc=0xdcd141
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5

Apr 30 '24 23:04 RichieSams

A new type of panic:

fatal error: runtime: name offset out of range
goroutine 1 [running, locked to thread]:
runtime.throw({0x187d65c?, 0x1850bb5?})
	runtime/panic.go:1077 +0x65 fp=0xc0000fbcf8 sp=0xc0000fbcc8 pc=0x68c7a5
runtime.resolveNameOff(0x284d14a?, 0x6c5160)
	runtime/type.go:119 +0x245 fp=0xc0000fbd50 sp=0xc0000fbcf8 pc=0x6b1765
internal/reflectlite.resolveNameOff(0x0?, 0x0?)
	runtime/runtime1.go:626 +0x13 fp=0xc0000fbd70 sp=0xc0000fbd50 pc=0x6b82d3
internal/reflectlite.rtype.nameOff(...)
	internal/reflectlite/type.go:244
internal/reflectlite.rtype.String({0xc000218030?})
	internal/reflectlite/type.go:256 +0x1b fp=0xc0000fbd98 sp=0xc0000fbd70 pc=0x6c4d7b
internal/reflectlite.elem(0xc0000fbdf0?)
	internal/reflectlite/type.go:320 +0x73 fp=0xc0000fbdd0 sp=0xc0000fbd98 pc=0x6c5113
internal/reflectlite.rtype.Elem({0xc000218000?})
	internal/reflectlite/type.go:324 +0x13 fp=0xc0000fbde8 sp=0xc0000fbdd0 pc=0x6c5173
errors.init()
	errors/wrap.go:136 +0x43 fp=0xc0000fbe10 sp=0xc0000fbde8 pc=0x6c90c3
runtime.doInit1(0x24bccc0)
	runtime/proc.go:6740 +0xdd fp=0xc0000fbf40 sp=0xc0000fbe10 pc=0x69bf1d
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc0000fbfe0 sp=0xc0000fbf40 pc=0x68f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0x6bd241
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7fa8 sp=0xc0000f7f88 pc=0x68f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f7fe0 sp=0xc0000f7fa8 pc=0x68f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x6bd241
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f9f78 sp=0xc0000f9f58 pc=0x68f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f9fc8 sp=0xc0000f9f78 pc=0x678ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f9fe0 sp=0xc0000f9fc8 pc=0x66dc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0x6bd241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 18 [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x1ae7600?, 0x1?, 0x0?, 0xc0002041a0?)
	runtime/proc.go:398 +0xce fp=0xc00020df70 sp=0xc00020df50 pc=0x68f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x27ea440)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc00020dfa0 sp=0xc00020df70 pc=0x676369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc00020dfc8 sp=0xc00020dfa0 pc=0x6768fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc00020dfe0 sp=0xc00020dfc8 pc=0x66dbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc00020dfe8 sp=0xc00020dfe0 pc=0x6bd241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5

May 03 '24 20:05 RichieSams

NOTE: We compile inside docker. The output below is from running go env within the docker container that we use. We cross-compile for linux, windows, and darwin. The workers having a problem in this case are running windows/amd64.

These workers are bare metal AMD Threadripper machines running Windows 10.

Just to be clear, you're cross compiling inside docker, but running the program on real Windows machines? Do you see it only on a specific type of Windows machines? Thanks.

May 06 '24 18:05 cherrymui

Just to be clear, you're cross compiling inside docker, but running the program on real Windows machines?

Correct. We cross compile in docker. But we're running in native Windows.

Do you see it only on a specific type of Windows machines? Thanks.

Yes. We only see it on our AMD servers. Example specs of the machines is listed above in the original post. None of our Intel-based servers are seeing these panics

Let me know if you have any other questions. :)

May 06 '24 20:05 RichieSams

Thanks. Have you tried running the program under the race detector? Is the program using cgo? Is there a way we can reproduce the issue ourselves?

May 06 '24 20:05 cherrymui

cc @golang/windows

May 06 '24 20:05 cherrymui

Have you tried running the program under the race detector?

No, but that's a good idea. I'll give that a shot today

Is the program using cgo?

No, we disable it with CGO_ENABLED=0 when compiling

Is there a way we can reproduce the issue ourselves?

Unfortunately, it's a proprietary program. But given it's panic'ing on init() and in standard libraries, I'm going to see if I can create a small "hello world" test program and just run it continuously on the machines to see if I can reproduce it.

May 07 '24 12:05 RichieSams

I'm wondering if there is some small incorrect assumption with AMD hardware in the golang runtime code? I see this other issue, which isn't the same stack trace, but is also limited to AMD hardware: https://github.com/golang/go/issues/62440

May 07 '24 13:05 RichieSams

Some more callstacks from our servers:

fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0xccfa1
PC=0xccfa1
runtime.abort()
	runtime/asm_amd64.s:1113 +0x1 fp=0xc0000f9bc0 sp=0xc0000f9bb8 pc=0xccfa1
traceback: unexpected SPWRITE function runtime.morestack
runtime.morestack()
	runtime/asm_amd64.s:560 +0x24 fp=0xc0000f9bc8 sp=0xc0000f9bc0 pc=0xcb204
goroutine 1 [running, locked to thread]:
runtime.doInit1(0x1e82a00?)
	runtime/proc.go:6711 +0x4c5 fp=0xc0000f9f40 sp=0xc0000f9f38 pc=0xac305
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc0000f9fe0 sp=0xc0000f9f40 pc=0x9f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0xcd141
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f5fa8 sp=0xc0000f5f88 pc=0x9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f5fe0 sp=0xc0000f5fa8 pc=0x9f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0xcd141
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7f78 sp=0xc0000f7f58 pc=0x9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f7fc8 sp=0xc0000f7f78 pc=0x88ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f7fe0 sp=0xc0000f7fc8 pc=0x7dc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0xcd141
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000440e0?, 0x14b8428?, 0x1?, 0x0?, 0xc0000f2b60?)
	runtime/proc.go:398 +0xce fp=0xc000107f70 sp=0xc000107f50 pc=0x9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x2163e20)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc000107fa0 sp=0xc000107f70 pc=0x86369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc000107fc8 sp=0xc000107fa0 pc=0x868fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc000107fe0 sp=0xc000107fc8 pc=0x7dbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc000107fe8 sp=0xc000107fe0 pc=0xcd141
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5
rax     0x17
rbx     0x2164e80
rcx     0x2164c00
rdi     0xcc34a2e000
rsi     0xc0000f9ab8
rbp     0xc0000f9bd8
rsp     0xc0000f9bb8
r8      0xc0000f9888
r9      0xc0000f9a60
r10     0x0
r11     0x246
r12     0xcc34dff6d0
r13     0x1000
r14     0x21645e0
r15     0xb2c
rip     0xccfa1
rflags  0x212
cs      0x33
fs      0x53
gs      0x2b

panic: godebug: Value of name not listed in godebugs.All: 
goroutine 1 [running]:
internal/godebug.(*Setting).Value.func1()
	internal/godebug/godebug.go:141 +0xb9
sync.(*Once).doSlow(0x28ba410?, 0xc0000f9dd8?)
	sync/once.go:74 +0xbf
sync.(*Once).Do(...)
	sync/once.go:65
internal/godebug.(*Setting).Value(0x28ba410)
	internal/godebug/godebug.go:138 +0x48
internal/intern.safeMap(...)
	internal/intern/intern.go:74
internal/intern.init()
	internal/intern/intern.go:66 +0x48

May 07 '24 14:05 RichieSams

Another callstack:

panic: runtime error: makeslice: cap out of range
goroutine 1 [running]:
runtime/metrics.init.0()
	runtime/metrics/description.go:457 +0xa6

May 08 '24 14:05 RichieSams

fatal error: fault
[signal 0xc0000005 code=0x0 addr=0x1f800608 pc=0x142df7e]
goroutine 1 gp=0xc0000fc000 m=0 mp=0x3240880 [running, locked to thread]:
runtime.throw({0x22296ae?, 0xfe15f40ffe15f40f?})
	runtime/panic.go:1023 +0x65 fp=0xc00019d798 sp=0xc00019d768 pc=0x102ebc5
runtime.sigpanic()
	runtime/signal_windows.go:414 +0xd0 fp=0xc00019d7e0 sp=0xc00019d798 pc=0x1044670
golang.org/x/text/internal/language/compact.getCoreIndex.func1(0x249?)
	golang.org/x/[email protected]/internal/language/compact/compact.go:32 +0x1e fp=0xc00019d800 sp=0xc00019d7e0 pc=0x142df7e
sort.Search(0xc0001ee060?, 0xc00019d848)
	sort/search.go:65 +0x46 fp=0xc00019d830 sp=0xc00019d800 pc=0x109db26
golang.org/x/text/internal/language/compact.getCoreIndex({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
	golang.org/x/[email protected]/internal/language/compact/compact.go:31 +0x9f fp=0xc00019d888 sp=0xc00019d830 pc=0x142debf
golang.org/x/text/internal/language/compact.FromTag({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
	golang.org/x/[email protected]/internal/language/compact/language.go:238 +0x5a5 fp=0xc00019db90 sp=0xc00019d888 pc=0x142f145
golang.org/x/text/internal/language/compact.Make({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
	golang.org/x/[email protected]/internal/language/compact/language.go:53 +0x285 fp=0xc00019dc30 sp=0xc00019db90 pc=0x142e505
golang.org/x/text/language.makeTag(...)
	golang.org/x/[email protected]/language/language.go:25
golang.org/x/text/language.CanonType.Parse(0x17, {0x2241259?, 0xc0001bc070?})
	golang.org/x/[email protected]/language/parse.go:62 +0x1f7 fp=0xc00019dd58 sp=0xc00019dc30 pc=0x14343d7
golang.org/x/text/language.Parse(...)
	golang.org/x/[email protected]/language/parse.go:35
golang.org/x/text/language.MustParse({0x2241259?, 0x15?})
	golang.org/x/[email protected]/language/tags.go:14 +0x25 fp=0xc00019dd80 sp=0xc00019dd58 pc=0x1434645
golang.org/x/text/cases.init.0()
	golang.org/x/[email protected]/cases/map.go:43 +0xa8 fp=0xc00019de20 sp=0xc00019dd80 pc=0x143aaa8
runtime.doInit1(0x2f02670)
	runtime/proc.go:7176 +0xed fp=0xc00019df50 sp=0xc00019de20 pc=0x103f82d
runtime.doInit(...)
	runtime/proc.go:7143
runtime.main()
	runtime/proc.go:253 +0x327 fp=0xc00019dfe0 sp=0xc00019df50 pc=0x1031587
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc00019dfe8 sp=0xc00019dfe0 pc=0x1062801
goroutine 18 gp=0xc0001841c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc000101fa8 sp=0xc000101f88 pc=0x10318ee
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.forcegchelper()
	runtime/proc.go:326 +0xb8 fp=0xc000101fe0 sp=0xc000101fa8 pc=0x1031778
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000101fe8 sp=0xc000101fe0 pc=0x1062801
created by runtime.init.6 in goroutine 1
	runtime/proc.go:314 +0x1a
goroutine 2 gp=0xc0000fca80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc00010ff80 sp=0xc00010ff60 pc=0x10318ee
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.bgsweep(0xc0000480e0)
	runtime/mgcsweep.go:278 +0x94 fp=0xc00010ffc8 sp=0xc00010ff80 pc=0x101ab14
runtime.gcenable.gowrap1()
	runtime/mgc.go:203 +0x25 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x100f405
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x1062801
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:203 +0x66
goroutine 3 gp=0xc0000fcc40 m=nil [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x2508298?, 0x1?, 0x0?, 0xc0000fcc40?)
	runtime/proc.go:402 +0xce fp=0xc000111f78 sp=0xc000111f58 pc=0x10318ee
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.(*scavengerState).park(0x323ec40)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc000111fa8 sp=0xc000111f78 pc=0x10184e9
runtime.bgscavenge(0xc0000480e0)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc000111fc8 sp=0xc000111fa8 pc=0x1018a7c
runtime.gcenable.gowrap2()
	runtime/mgc.go:204 +0x25 fp=0xc000111fe0 sp=0xc000111fc8 pc=0x100f3a5
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x1062801
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:204 +0xa5
goroutine 4 gp=0xc0000fce00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc00010be20 sp=0xc00010be00 pc=0x10318ee
runtime.runfinq()
	runtime/mfinal.go:194 +0x107 fp=0xc00010bfe0 sp=0xc00010be20 pc=0x100e487
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0x1062801
created by runtime.createfing in goroutine 1
	runtime/mfinal.go:164 +0x3d

May 08 '24 14:05 RichieSams

I'm wondering if there is some small incorrect assumption with AMD hardware in the golang runtime code? I see this other issue, which isn't the same stack trace, but is also limited to AMD hardware: https://github.com/golang/go/issues/62440

It could be. It could also be a kernel issue. A reproducer would be very helpful. Thanks.

May 10 '24 15:05 cherrymui

I'm attempting to get a minimal repro. I created something and I'm running it on all our workers periodically to try to get a failure. I'll report back here if I can get it to trigger.

May 17 '24 18:05 RichieSams

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

Aug 14 '24 22:08 gopherbot

I am experiencing this issue and can reproduce. It occurs intermittently when I attempt to use backup tool restic. Let me know what needs to be done and I will see if I can help.

My hardware is: AMD EPYC 7543P 32-Core Processor

PS C:\restic> .\restic_0.17.0_windows_amd64.exe
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x18d2ad1?, 0x16fd540?})
        /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc00035fba0 sp=0xc00035fb70 pc=0xa6d405
runtime.badmcall(0xc00035fbf8?)
        /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc00035fbc0 sp=0xc00035fba0 pc=0xa707bf
runtime.badmcall(0xa6e899)
        <autogenerated>:1 +0x25 fp=0xc00035fbd8 sp=0xc00035fbc0 pc=0xaa5be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00035fbf0
stack: frame={sp:0xc00035fbc0, fp:0xc00035fbd8} stack=[0x15ed004000,0x15ed1ff850)


goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
time.NewTicker(0x2540be400?)
        /usr/local/go/src/time/tick.go:20 +0xf0 fp=0xc00035fd90 sp=0xc00035fd88 pc=0xaccc90
go.opencensus.io/stats/view.NewMeter()
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:266 +0x45 fp=0xc00035fdf0 sp=0xc00035fd90 pc=0x1444b05
go.opencensus.io/stats/view.init.0()
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:33 +0x17 fp=0xc00035fe20 sp=0xc00035fdf0 pc=0x1444037
runtime.doInit1(0x2253870)
        /usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc00035ff50 sp=0xc00035fe20 pc=0xa7e06d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7143
runtime.main()
        /usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc00035ffe0 sp=0xc00035ff50 pc=0xa6fdc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00035ffe8 sp=0xc00035ffe0 pc=0xaa0ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xa7012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xa6ffb8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xaa0ea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xa7012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
        /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xa5a2d4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xa4eb45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xaa0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x1a7dd08?, 0x1?, 0x0?, 0xc000066c40?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0xa7012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x22f43c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0xa57cc9
runtime.bgscavenge(0xc000074000)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0xa5825c
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0xa4eae5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xaa0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 18 gp=0xc0001041c0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0xa570fc?, 0xc0?, 0xe5?, 0x550011?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0xa7012e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0xa4dbc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xaa0ea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:164 +0x3d
PS C:\restic>

Aug 21 '24 03:08 distancesprinter

@distancesprinter Is there a way that we can reproduce the problem ourselves?

Aug 21 '24 03:08 ianlancetaylor

You will have to let me know what you mean by that. You could try to download restic_0.17.0_windows_amd64 and try to run it. It seems like my issues only occur after a certain hour of the day, but I am still trying to decipher a pattern.

On Tue, Aug 20, 2024, 11:16 PM Ian Lance Taylor @.***> wrote:

@distancesprinter https://github.com/distancesprinter Is there a way that we can reproduce the problem ourselves?

— Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/67108#issuecomment-2300496792, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADFATVEAYIHWNADJXUM4VVLZSQA67AVCNFSM6AAAAABG6TTIXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBQGQ4TMNZZGI . You are receiving this because you were mentioned.Message ID: @.***>

Aug 21 '24 03:08 distancesprinter

What I mean is a precise set of instructions that let us reliably reproduce the problem on our own systems. It sounds like we are not there yet.

Aug 21 '24 03:08 ianlancetaylor

Just so you are aware, I am not a Go developer, I only arrived here by Googling my error, but I will try to be as helpful as possible.

The software I am running is restic_0.17.0_windows_amd64.exe

restic 0.17.0 compiled with go1.22.5 on windows/amd64

There is build process information listed at the following locations: https://restic.readthedocs.io/en/stable/developer_information.html https://github.com/restic/builder

To be clear, I did not build the software. I am using the official release binary.

You can download the binary from: https://github.com/restic/restic/releases/tag/v0.17.0 https://github.com/restic/restic/releases/download/v0.17.0/restic_0.17.0_windows_amd64.zip

Extract the zip to a folder; I am using C:\restic

From Windows GUI, right-click Start and choose Windows PowerShell (Admin) to launch PowerShell as an Administrator.

Then cd to C:\restic and attempt to run the binary.

I have found that after booting the OS, if I repeatedly try to run the binary, eventually it works (usually on the 5th or 6th attempt).

I am running Windows Server 2019 Standard (Desktop Experience) in a bhyve VM.

Here is some console output, showing repeated attempts to execute the binary, and eventually the expected output. I also printed systeminfo so you have detailed build and hotfix information.

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> cd C:\restic\
PS C:\restic> .\restic_0.17.0_windows_amd64.exe
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1722ad1?, 0x10?})
        /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc000145c30 sp=0xc000145c00 pc=0x8bd405
runtime.badmcall(0x89a265?)
        /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc000145c50 sp=0xc000145c30 pc=0x8c07bf
runtime.badmcall(0x8be899)
        <autogenerated>:1 +0x25 fp=0xc000145c68 sp=0xc000145c50 pc=0x8f5be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000145c80
stack: frame={sp:0xc000145c50, fp:0xc000145c68} stack=[0x7d0ec04000,0x7d0edff710)


goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
golang.org/x/text/encoding/unicode.init()
        <autogenerated>:1 +0x1c6 fp=0xc000145e20 sp=0xc000145e18 pc=0xe92906
runtime.doInit1(0x20a1ed0)
        /usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc000145f50 sp=0xc000145e20 pc=0x8ce06d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7143
runtime.main()
        /usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc000145fe0 sp=0xc000145f50 pc=0x8bfdc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000145fe8 sp=0xc000145fe0 pc=0x8f0ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0x8bffb8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x8f0ea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
        /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0x8aa2d4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0x89eb45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 18 gp=0xc0001041c0 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x18cdd08?, 0x1?, 0x0?, 0xc0001041c0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00010df78 sp=0xc00010df58 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x21443c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00010dfa8 sp=0xc00010df78 pc=0x8a7cc9
runtime.bgscavenge(0xc000074000)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00010dfc8 sp=0xc00010dfa8 pc=0x8a825c
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00010dfe0 sp=0xc00010dfc8 pc=0x89eae5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 19 gp=0xc000104380 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0x8a70fc?, 0xc0?, 0xe5?, 0x550011?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0x8c012e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0x89dbc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x8f0ea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:164 +0x3d
PS C:\restic> .\restic_0.17.0_windows_amd64.exe
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1722ad1?, 0xc0003dfc60?})
        /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc0003dfc30 sp=0xc0003dfc00 pc=0x8bd405
runtime.badmcall(0xa300000002145fc0?)
        /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc0003dfc50 sp=0xc0003dfc30 pc=0x8c07bf
runtime.badmcall(0x8be899)
        <autogenerated>:1 +0x25 fp=0xc0003dfc68 sp=0xc0003dfc50 pc=0x8f5be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc0003dfc80
stack: frame={sp:0xc0003dfc50, fp:0xc0003dfc68} stack=[0x5de7604000,0x5de77ffd20)


goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
github.com/restic/restic/internal/feature.init.0()
        /restic/internal/feature/registry.go:16 +0x53e fp=0xc0003dfe20 sp=0xc0003dfe18 pc=0xc3745e
runtime.doInit1(0x20a1c70)
        /usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc0003dff50 sp=0xc0003dfe20 pc=0x8ce06d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7143
runtime.main()
        /usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc0003dffe0 sp=0xc0003dff50 pc=0x8bfdc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0003dffe8 sp=0xc0003dffe0 pc=0x8f0ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0x8bffb8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x8f0ea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
        /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0x8aa2d4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0x89eb45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x18cdd08?, 0x1?, 0x0?, 0xc000066c40?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x21443c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0x8a7cc9
runtime.bgscavenge(0xc000074000)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0x8a825c
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0x89eae5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0x8a70fc?, 0xc0?, 0xe5?, 0x550011?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0x8c012e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0x89dbc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x8f0ea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 18 gp=0xc000211500 m=nil [select]:
runtime.gopark(0xc000077f78?, 0x3?, 0x8?, 0x1?, 0xc000077f72?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077e18 sp=0xc000077df8 pc=0x8c012e
runtime.selectgo(0xc000077f78, 0xc000077f6c, 0xc000220d00?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000077f38 sp=0xc000077e18 pc=0x8d0585
go.opencensus.io/stats/view.(*worker).start(0xc000220d00)
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f fp=0xc000077fc8 sp=0xc000077f38 pc=0x1294dff
go.opencensus.io/stats/view.init.0.gowrap1()
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x25 fp=0xc000077fe0 sp=0xc000077fc8 pc=0x1294165
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x8f0ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d
PS C:\restic> .\restic_0.17.0_windows_amd64.exe
fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0x8f0d01
PC=0x8f0d01

runtime.abort()
        /usr/local/go/src/runtime/asm_amd64.s:1157 +0x1 fp=0xc00063fc20 sp=0xc00063fc18 pc=0x8f0d01
runtime.morestack()
        /usr/local/go/src/runtime/asm_amd64.s:593 +0x3d fp=0xc00063fc28 sp=0xc00063fc20 pc=0x8eef7d

goroutine 1 gp=0xc000066000 m=nil [runnable]:
github.com/spf13/cobra.(*Command).ExecuteC(0x21345e0)
        /home/build/go/pkg/mod/github.com/spf13/[email protected]/command.go:1054 +0x625 fp=0xc00063fe00 sp=0xc00063fdf8 pc=0xc28b05
github.com/spf13/cobra.(*Command).Execute(...)
        /home/build/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/build/go/pkg/mod/github.com/spf13/[email protected]/command.go:1034
main.main()
        /restic/cmd/restic/main.go:125 +0x265 fp=0xc00063ff50 sp=0xc00063fe00 pc=0x13ca3e5
runtime.main()
        /usr/local/go/src/runtime/proc.go:271 +0x28b fp=0xc00063ffe0 sp=0xc00063ff50 pc=0x8bfd2b
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00063ffe8 sp=0xc00063ffe0 pc=0x8f0ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0x8bffb8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x8f0ea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
        /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0x8aa2d4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0x89eb45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x18cdd08?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x21443c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0x8a7cc9
runtime.bgscavenge(0xc000074000)
        /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0x8a8279
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0x89eae5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0x8a70fc?, 0xc0?, 0xe5?, 0x550011?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0x8c012e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0x89dbc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x8f0ea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 6 gp=0xc0001b76c0 m=nil [select]:
runtime.gopark(0xc00007df78?, 0x3?, 0x98?, 0x5?, 0xc00007df72?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007de18 sp=0xc00007ddf8 pc=0x8c012e
runtime.selectgo(0xc00007df78, 0xc00007df6c, 0xc0000e5700?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00007df38 sp=0xc00007de18 pc=0x8d0585
go.opencensus.io/stats/view.(*worker).start(0xc0000e5700)
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f fp=0xc00007dfc8 sp=0xc00007df38 pc=0x1294dff
go.opencensus.io/stats/view.init.0.gowrap1()
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 pc=0x1294165
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x8f0ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 18 gp=0xc0004041c0 m=nil [chan receive]:
runtime.gopark(0x955c05?, 0x154a9c0?, 0x1?, 0x80?, 0xc000077f60?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077ed0 sp=0xc000077eb0 pc=0x8c012e
runtime.chanrecv(0xc00010c120, 0xc000077fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000077f48 sp=0xc000077ed0 pc=0x88aa0d
runtime.chanrecv2(0xc00010c120?, 0xc000077fc0?)
        /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc000077f70 sp=0xc000077f48 pc=0x88a632
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:404 +0x99 fp=0xc000077fe0 sp=0xc000077f70 pc=0xcd92f9
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x8f0ea1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
        /restic/internal/restic/lock.go:401 +0x1a

goroutine 34 gp=0xc0001041c0 m=6 mp=0xc000400008 [syscall]:
runtime.notetsleepg(0x21cd9a0, 0xffffffffffffffff)
        /usr/local/go/src/runtime/lock_sema.go:296 +0x31 fp=0xc000117fa0 sp=0xc000117f68 pc=0x890771
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000117fc0 sp=0xc000117fa0 pc=0x8ec789
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000117fe0 sp=0xc000117fc0 pc=0x955fd3
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0x8f0ea1
created by os/signal.Notify.func1.1 in goroutine 18
        /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 19 gp=0xc000405dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x9a1dd2f48?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000079f50 sp=0xc000079f30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000079fe0 sp=0xc000079f50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 50 gp=0xc000580000 m=nil [GC worker (idle)]:
runtime.gopark(0x9a1d1df94?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000113f50 sp=0xc000113f30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000113fe0 sp=0xc000113f50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000113fe8 sp=0xc000113fe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 7 gp=0xc0001b7a40 m=9 mp=0xc0003a6008 [running]:
        goroutine running on other thread; stack unavailable
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 20 gp=0xc000536000 m=nil [GC worker (idle)]:
runtime.gopark(0x9a1dd2f48?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00039ff50 sp=0xc00039ff30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00039ffe0 sp=0xc00039ff50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00039ffe8 sp=0xc00039ffe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 21 gp=0xc0005361c0 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0003a1e98 sp=0xc0003a1e78 pc=0x8c012e
runtime.chanrecv(0xc00041f620, 0xc0003a1fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc0003a1f10 sp=0xc0003a1e98 pc=0x88aa0d
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc0003a1f38 sp=0xc0003a1f10 pc=0x88a612
main.cleanupHandler(0x0?, 0xc000411950)
        /restic/cmd/restic/cleanup.go:24 +0x2a fp=0xc0003a1fc0 sp=0xc0003a1f38 pc=0x138f8aa
main.createGlobalContext.gowrap1()
        /restic/cmd/restic/cleanup.go:16 +0x25 fp=0xc0003a1fe0 sp=0xc0003a1fc0 pc=0x138f845
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0003a1fe8 sp=0xc0003a1fe0 pc=0x8f0ea1
created by main.createGlobalContext in goroutine 1
        /restic/cmd/restic/cleanup.go:16 +0x98
rax     0x17
rbx     0xc0001003c0
rcx     0xc000100008
rdx     0x0
rdi     0xe653814000
rsi     0xc00063fb18
rbp     0xc00063fc38
rsp     0xc00063fc18
r8      0xc00063f8e8
r9      0xc00063fac0
r10     0x0
r11     0x246
r12     0xe6547ff7e0
r13     0x0
r14     0xc000104000
r15     0x1
rip     0x8f0d01
rflags  0x212
cs      0x33
fs      0x53
gs      0x2b
PS C:\restic> .\restic_0.17.0_windows_amd64.exe
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1722ad1?, 0x10?})
        /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc00035f050 sp=0xc00035f020 pc=0x8bd405
runtime.badmcall(0x18d1c720108?)
        /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc00035f070 sp=0xc00035f050 pc=0x8c07bf
runtime.badmcall(0x8be899)
        <autogenerated>:1 +0x25 fp=0xc00035f088 sp=0xc00035f070 pc=0x8f5be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00035f0a0
stack: frame={sp:0xc00035f070, fp:0xc00035f088} stack=[0x438da04000,0x438dbffca0)


goroutine 23 gp=0xc00046ea80 m=0 mp=0x2145fc0 [GC worker (active)]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc0004c3f50 sp=0xc0004c3f40 pc=0x8eee88
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1370 +0x1f7 fp=0xc0004c3fe0 sp=0xc0004c3f50 pc=0x8a0d97
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004c3fe8 sp=0xc0004c3fe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 1 gp=0xc000066000 m=nil [runnable]:
text/template.goodFunc({0x18f6b50, 0x14d9100})
        /usr/local/go/src/text/template/funcs.go:112 +0xac fp=0xc00063f238 sp=0xc00063f230 pc=0xbfd1ac
reflect.unpackEface(...)
        /usr/local/go/src/reflect/value.go:160
reflect.ValueOf(...)
        /usr/local/go/src/reflect/value.go:3264
text/template.addValueFuncs(0xc000046540, 0xc00035f390?)
        /usr/local/go/src/text/template/funcs.go:92 +0xf3 fp=0xc00063f360 sp=0xc00063f238 pc=0xbfce73
text/template.createValueFuncs(...)
        /usr/local/go/src/text/template/funcs.go:82
text/template.builtinFuncs.func1()
        /usr/local/go/src/text/template/funcs.go:74 +0x5ae fp=0xc00063f3d0 sp=0xc00063f360 pc=0xc036ce
sync.(*Once).doSlow(0x75?, 0xc000090000?)
        /usr/local/go/src/sync/once.go:74 +0xc2 fp=0xc00063f430 sp=0xc00063f3d0 pc=0x8feca2
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:65
text/template.builtinFuncs()
        /usr/local/go/src/text/template/funcs.go:73 +0x2c fp=0xc00063f450 sp=0xc00063f430 pc=0xbfcd4c
text/template.findFunction({0x174c033, 0x2}, 0xc00035f520?)
        /usr/local/go/src/text/template/funcs.go:149 +0x145 fp=0xc00063f4b8 sp=0xc00063f450 pc=0xbfd4c5
text/template.(*state).evalFunction(0xc00063fc00, {0x16eda80?, 0x21345e0?, 0x2?}, 0x47?, {0x18e87f0, 0xc0000461b0}, {0xc00007e040, 0x3, 0x4}, ...)
        /usr/local/go/src/text/template/exec.go:619 +0xa6 fp=0xc00063f5b0 sp=0xc00063f4b8 pc=0xbf8ec6
text/template.(*state).evalCommand(0xc00035fc00, {0x16eda80?, 0x21345e0?, 0x16ef3b6?}, 0xc0000461b0, {0x1512160?, 0x21cc760?, 0x8?})
        /usr/local/go/src/text/template/exec.go:510 +0x1ce fp=0xc00063f6a8 sp=0xc00063f5b0 pc=0xbf7fce
text/template.(*state).evalPipeline(0xc00063fc00, {0x16eda80?, 0x21345e0?, 0xc000058008?}, 0xc000080240)
        /usr/local/go/src/text/template/exec.go:479 +0x125 fp=0xc00063f7a8 sp=0xc00063f6a8 pc=0xbf7945
text/template.(*state).evalCommand(0xc00035fc00, {0x16eda80?, 0x21345e0?, 0xd0?}, 0xc000046180, {0x1512160?, 0x21cc760?, 0xc0000480d0?})
        /usr/local/go/src/text/template/exec.go:514 +0x269 fp=0xc00063f8a0 sp=0xc00063f7a8 pc=0xbf8069
text/template.(*state).evalPipeline(0xc00063fc00, {0x16eda80?, 0x21345e0?, 0x15e9580?}, 0xc0000801e0)
        /usr/local/go/src/text/template/exec.go:479 +0x125 fp=0xc00063f9a0 sp=0xc00063f8a0 pc=0xbf7945
text/template.(*state).walkIfOrWith(0xc00063fc00, 0x13, {0x16eda80?, 0x21345e0?, 0x89a53d?}, 0x18d1c72ac48?, 0xc000046270, 0x0)
        /usr/local/go/src/text/template/exec.go:301 +0xc6 fp=0xc00063fa60 sp=0xc00063f9a0 pc=0xbf63a6
text/template.(*state).walk(0xc00035fc00, {0x16eda80?, 0x21345e0?, 0x18d1c72ac48?}, {0x18e8ac0?, 0xc00007e080?})
        /usr/local/go/src/text/template/exec.go:291 +0x1bb fp=0xc00063faf8 sp=0xc00063fa60 pc=0xbf60db
text/template.(*state).walk(0xc00063fc00, {0x16eda80?, 0x21345e0?, 0xc000092120?}, {0x18e8880?, 0xc000046150?})
        /usr/local/go/src/text/template/exec.go:280 +0x2e5 fp=0xc00063fb90 sp=0xc00063faf8 pc=0xbf6205
text/template.(*Template).execute(0xc000437100, {0x18d6860?, 0xc00010c058?}, {0x16eda80?, 0x21345e0?})
        /usr/local/go/src/text/template/exec.go:224 +0x278 fp=0xc00063fc80 sp=0xc00063fb90 pc=0xbf5e38
text/template.(*Template).Execute(...)
        /usr/local/go/src/text/template/exec.go:207
github.com/spf13/cobra.tmpl({0x18d6860, 0xc00010c058}, {0x174c02b, 0x7c}, {0x16eda80, 0x21345e0})
        /home/build/go/pkg/mod/github.com/spf13/[email protected]/cobra.go:184 +0x165 fp=0xc00063fcc8 sp=0xc00063fc80 pc=0xc241a5
github.com/spf13/cobra.(*Command).HelpFunc.func1(0x21345e0, {0x2103460?, 0x18d6a60?, 0x2103460?})
        /home/build/go/pkg/mod/github.com/spf13/[email protected]/command.go:465 +0x69 fp=0xc00063fd28 sp=0xc00063fcc8 pc=0xc355e9
github.com/spf13/cobra.(*Command).ExecuteC(0x21345e0)
        /home/build/go/pkg/mod/github.com/spf13/[email protected]/command.go:1122 +0x5d7 fp=0xc00063fe00 sp=0xc00063fd28 pc=0xc28ab7
github.com/spf13/cobra.(*Command).Execute(...)
        /home/build/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/build/go/pkg/mod/github.com/spf13/[email protected]/command.go:1034
main.main()
        /restic/cmd/restic/main.go:125 +0x265 fp=0xc00063ff50 sp=0xc00063fe00 pc=0x13ca3e5
runtime.main()
        /usr/local/go/src/runtime/proc.go:271 +0x28b fp=0xc00063ffe0 sp=0xc00063ff50 pc=0x8bfd2b
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00063ffe8 sp=0xc00063ffe0 pc=0x8f0ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0x8bffb8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x8f0ea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
        /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0x8aa2d4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0x89eb45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x18cdd08?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x21443c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0x8a7cc9
runtime.bgscavenge(0xc000074000)
        /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0x8a8279
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0x89eae5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 18 gp=0xc0001041c0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0x8a70fc?, 0xc0?, 0xe5?, 0x550011?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0x8c012e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0x89dbc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x8f0ea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 19 gp=0xc0002168c0 m=nil [select]:
runtime.gopark(0xc000077f78?, 0x3?, 0x8?, 0xca?, 0xc000077f72?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077e18 sp=0xc000077df8 pc=0x8c012e
runtime.selectgo(0xc000077f78, 0xc000077f6c, 0xc000193700?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000077f38 sp=0xc000077e18 pc=0x8d0585
go.opencensus.io/stats/view.(*worker).start(0xc000193700)
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f fp=0xc000077fc8 sp=0xc000077f38 pc=0x1294dff
go.opencensus.io/stats/view.init.0.gowrap1()
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x25 fp=0xc000077fe0 sp=0xc000077fc8 pc=0x1294165
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x8f0ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 20 gp=0xc000216a80 m=nil [chan receive]:
runtime.gopark(0x955be5?, 0x154a9c0?, 0x1?, 0x0?, 0xc000079f60?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000079ed0 sp=0xc000079eb0 pc=0x8c012e
runtime.chanrecv(0xc000080060, 0xc000079fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000079f48 sp=0xc000079ed0 pc=0x88aa0d
runtime.chanrecv2(0xc000080060?, 0xc000079fc0?)
        /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc000079f70 sp=0xc000079f48 pc=0x88a632
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:404 +0x99 fp=0xc000079fe0 sp=0xc000079f70 pc=0xcd92f9
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x8f0ea1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
        /restic/internal/restic/lock.go:401 +0x1a

goroutine 21 gp=0xc00046e700 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0004c7e98 sp=0xc0004c7e78 pc=0x8c012e
runtime.chanrecv(0xc00041d0e0, 0xc0004c7fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc0004c7f10 sp=0xc0004c7e98 pc=0x88aa0d
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc0004c7f38 sp=0xc0004c7f10 pc=0x88a612
main.cleanupHandler(0x0?, 0xc000444ae0)
        /restic/cmd/restic/cleanup.go:24 +0x2a fp=0xc0004c7fc0 sp=0xc0004c7f38 pc=0x138f8aa
main.createGlobalContext.gowrap1()
        /restic/cmd/restic/cleanup.go:16 +0x25 fp=0xc0004c7fe0 sp=0xc0004c7fc0 pc=0x138f845
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004c7fe8 sp=0xc0004c7fe0 pc=0x8f0ea1
created by main.createGlobalContext in goroutine 1
        /restic/cmd/restic/cleanup.go:16 +0x98

goroutine 22 gp=0xc00046e8c0 m=6 mp=0xc000084008 [syscall]:
runtime.notetsleepg(0x21cd9a0, 0xffffffffffffffff)
        /usr/local/go/src/runtime/lock_sema.go:296 +0x31 fp=0xc0004c9fa0 sp=0xc0004c9f68 pc=0x890771
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc0004c9fc0 sp=0xc0004c9fa0 pc=0x8ec789
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0004c9fe0 sp=0xc0004c9fc0 pc=0x955fd3
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004c9fe8 sp=0xc0004c9fe0 pc=0x8f0ea1
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 5 gp=0xc000067180 m=nil [GC worker (idle)]:
runtime.gopark(0x21ce5c0?, 0x3?, 0x4?, 0xe2?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007df50 sp=0xc00007df30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00007dfe0 sp=0xc00007df50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 34 gp=0xc000500000 m=nil [GC worker (idle)]:
runtime.gopark(0x9f5f67c74?, 0x3?, 0xe0?, 0x88?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000507f50 sp=0xc000507f30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000507fe0 sp=0xc000507f50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 50 gp=0xc000582000 m=nil [GC worker (idle)]:
runtime.gopark(0x9f5f67c74?, 0xc000580020?, 0x1a?, 0xa?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000503f50 sp=0xc000503f30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000503fe0 sp=0xc000503f50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000503fe8 sp=0xc000503fe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c
PS C:\restic> .\restic_0.17.0_windows_amd64.exe

restic is a backup program which allows saving multiple revisions of files and
directories in an encrypted repository stored on different backends.

The full documentation can be found at https://restic.readthedocs.io/ .

Usage:
  restic [command]

Available Commands:
  backup        Create a new backup of files and/or directories
  cache         Operate on local cache directories
  cat           Print internal objects to stdout
  check         Check the repository for errors
  copy          Copy snapshots from one repository to another
  diff          Show differences between two snapshots
  dump          Print a backed-up file to stdout
  find          Find a file, a directory or restic IDs
  forget        Remove snapshots from the repository
  generate      Generate manual pages and auto-completion files (bash, fish, zsh, powershell)
  help          Help about any command
  init          Initialize a new repository
  key           Manage keys (passwords)
  list          List objects in the repository
  ls            List files in a snapshot
  migrate       Apply migrations
  prune         Remove unneeded data from the repository
  recover       Recover data from the repository not referenced by snapshots
  repair        Repair the repository
  restore       Extract the data from a snapshot
  rewrite       Rewrite snapshots to exclude unwanted files
  self-update   Update the restic binary
  snapshots     List all snapshots
  stats         Scan the repository and show basic statistics
  tag           Modify tags on snapshots
  unlock        Remove locks other processes created
  version       Print version information

Flags:
      --cacert file                file to load root certificates from (default: use system certificates or $RESTIC_CACERT)
      --cache-dir directory        set the cache directory. (default: use system default cache directory)
      --cleanup-cache              auto remove old cache directories
      --compression mode           compression mode (only available for repository format version 2), one of (auto|off|max) (default: $RESTIC_COMPRESSION) (default auto)
  -h, --help                       help for restic
      --http-user-agent string     set a http user agent for outgoing http requests
      --insecure-no-password       use an empty password for the repository, must be passed to every restic command (insecure)
      --insecure-tls               skip TLS certificate verification when connecting to the repository (insecure)
      --json                       set output mode to JSON for commands that support it
      --key-hint key               key ID of key to try decrypting first (default: $RESTIC_KEY_HINT)
      --limit-download rate        limits downloads to a maximum rate in KiB/s. (default: unlimited)
      --limit-upload rate          limits uploads to a maximum rate in KiB/s. (default: unlimited)
      --no-cache                   do not use a local cache
      --no-extra-verify            skip additional verification of data before upload (see documentation)
      --no-lock                    do not lock the repository, this allows some operations on read-only repositories
  -o, --option key=value           set extended option (key=value, can be specified multiple times)
      --pack-size size             set target pack size in MiB, created pack files may be larger (default: $RESTIC_PACK_SIZE)
      --password-command command   shell command to obtain the repository password from (default: $RESTIC_PASSWORD_COMMAND)
  -p, --password-file file         file to read the repository password from (default: $RESTIC_PASSWORD_FILE)
  -q, --quiet                      do not output comprehensive progress report
  -r, --repo repository            repository to backup to or restore from (default: $RESTIC_REPOSITORY)
      --repository-file file       file to read the repository location from (default: $RESTIC_REPOSITORY_FILE)
      --retry-lock duration        retry to lock the repository if it is already locked, takes a value like 5m or 2h (default: no retries)
      --tls-client-cert file       path to a file containing PEM encoded TLS client certificate and private key (default: $RESTIC_TLS_CLIENT_CERT)
  -v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)

Use "restic [command] --help" for more information about a command.
PS C:\restic> systeminfo /fo csv | ConvertFrom-Csv | select OS*, System*, Hotfix* | Format-List


OS Name             : Microsoft Windows Server 2019 Standard
OS Version          : 10.0.17763 N/A Build 17763
OS Manufacturer     : Microsoft Corporation
OS Configuration    : Member Server
OS Build Type       : Multiprocessor Free
System Boot Time    : 8/21/2024, 12:16:08 AM
System Manufacturer : FreeBSD
System Model        : BHYVE
System Type         : x64-based PC
System Directory    : C:\Windows\system32
System Locale       : en-us;English (United States)
Hotfix(s)           : 5 Hotfix(s) Installed.,[01]: KB5041913,[02]: KB4577667,[03]: KB4589208,[04]: KB5041578,[05]:
                      KB5041577



PS C:\restic> .\restic_0.17.0_windows_amd64.exe version
restic 0.17.0 compiled with go1.22.5 on windows/amd64

Aug 21 '24 04:08 distancesprinter

Thanks. CC @golang/windows

Aug 21 '24 17:08 ianlancetaylor

Contributing some additional information:

I recompiled restic from source with go 1.23.0 (in docker, debian-stable as per the restic build instructions).

Here is the output following a reboot when trying to print the version string--three panics before a success:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> cd C:\restic\
PS C:\restic> .\restic_0.17.0_windows_amd64_g123.exe version
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1acd5ed?, 0x0?})
        /usr/local/go/src/runtime/panic.go:1067 +0x4d fp=0xc0000517d8 sp=0xc0000517a8 pc=0xc4ea6d
runtime.badmcall(0x0?)
        /usr/local/go/src/runtime/proc.go:539 +0x1f fp=0xc0000517f8 sp=0xc0000517d8 pc=0xc1daff
runtime.badmcall(0xc1bc99)
        <autogenerated>:1 +0x25 fp=0xc000051810 sp=0xc0000517f8 pc=0xc5bf25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000051828
stack: frame={sp:0xc0000517f8, fp:0xc000051810} stack=[0x4a4a404000,0x4a4a5ff710)


goroutine 1 gp=0xc000068000 m=nil [runnable, locked to thread]:
sync.(*RWMutex).Lock(0x256fdb0?)
        /usr/local/go/src/sync/rwmutex.go:142 +0x7a fp=0xc0000519c8 sp=0xc0000519c0 pc=0xc676da
google.golang.org/protobuf/reflect/protoregistry.(*Types).RegisterEnum(0xc000120990, {0x1c8e450, 0xc000112d80})
        /home/build/go/pkg/mod/google.golang.org/[email protected]/reflect/protoregistry/registry.go:522 +0x79 fp=0xc000051a58 sp=0xc0000519c8 pc=0x1451c59
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x18264e9, 0x37}, {0x248fce0, 0x226, 0x226}, 0x2, 0x2, 0x0, 0x0, {0x0, ...}, ...}, ...})
        /home/build/go/pkg/mod/google.golang.org/[email protected]/internal/filetype/build.go:153 +0x169e fp=0xc000051c30 sp=0xc000051a58 pc=0x14e12fe
github.com/google/s2a-go/internal/proto/common_go_proto.file_internal_proto_common_common_proto_init()
        /home/build/go/pkg/mod/github.com/google/[email protected]/internal/proto/common_go_proto/common.pb.go:384 +0x278 fp=0xc000051e10 sp=0xc000051c30 pc=0x15b3e38
github.com/google/s2a-go/internal/proto/common_go_proto.init.0()
        /home/build/go/pkg/mod/github.com/google/[email protected]/internal/proto/common_go_proto/common.pb.go:344 +0xf fp=0xc000051e20 sp=0xc000051e10 pc=0x15b3baf
runtime.doInit1(0x2484970)
        /usr/local/go/src/runtime/proc.go:7278 +0xed fp=0xc000051f50 sp=0xc000051e20 pc=0xc2b10d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7245
runtime.main()
        /usr/local/go/src/runtime/proc.go:254 +0x325 fp=0xc000051fe0 sp=0xc000051f50 pc=0xc1d1a5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0xc571e1

goroutine 2 gp=0xc000068700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006bfa8 sp=0xc00006bf88 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:337 +0xb8 fp=0xc00006bfe0 sp=0xc00006bfa8 pc=0xc1d418
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xc571e1
created by runtime.init.7 in goroutine 1
        /usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 18 gp=0xc0001061c0 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00010ff80 sp=0xc00010ff60 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000078000)
        /usr/local/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00010ffc8 sp=0xc00010ff80 pc=0xc06334
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0xbfa965
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 19 gp=0xc000106380 m=nil [GC scavenge wait]:
runtime.gopark(0xc000078000?, 0x1c7a718?, 0x1?, 0x0?, 0xc000106380?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000111f78 sp=0xc000111f58 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x25264c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000111fa8 sp=0xc000111f78 pc=0xc03d69
runtime.bgscavenge(0xc000078000)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000111fc8 sp=0xc000111fa8 pc=0xc042dc
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000111fe0 sp=0xc000111fc8 pc=0xbfa905
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 34 gp=0xc0000861c0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00010be20 sp=0xc00010be00 pc=0xc4eb8e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00010bfe0 sp=0xc00010be20 pc=0xbf9a27
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0xc571e1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:163 +0x3d
PS C:\restic> .\restic_0.17.0_windows_amd64_g123.exe version
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1acd5ed?, 0x3ff5554?})
        /usr/local/go/src/runtime/panic.go:1067 +0x4d fp=0xc000051bd8 sp=0xc000051ba8 pc=0xc4ea6d
runtime.badmcall(0xc000051c50?)
        /usr/local/go/src/runtime/proc.go:539 +0x1f fp=0xc000051bf8 sp=0xc000051bd8 pc=0xc1daff
runtime.badmcall(0xc1bc99)
        <autogenerated>:1 +0x25 fp=0xc000051c10 sp=0xc000051bf8 pc=0xc5bf25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000051c28
stack: frame={sp:0xc000051bf8, fp:0xc000051c10} stack=[0x7574c04000,0x7574dff900)


goroutine 1 gp=0xc000068000 m=nil [runnable, locked to thread]:
runtime.mapassign_faststr(0x18a9820?, 0xc00017dce0?, {0x1a9989c, 0x2})
        /usr/local/go/src/runtime/map_faststr.go:223 +0x3f5 fp=0xc000051dc0 sp=0xc000051db8 pc=0xc4cef5
regexp/syntax.map.init.0()
        /usr/local/go/src/regexp/syntax/perl_groups.go:26 +0x1c fp=0xc000051e10 sp=0xc000051dc0 pc=0xf910dc
regexp/syntax.init()
        /usr/local/go/src/regexp/syntax/perl_groups.go:26 +0xf fp=0xc000051e20 sp=0xc000051e10 pc=0xf910af
runtime.doInit1(0x2483140)
        /usr/local/go/src/runtime/proc.go:7278 +0xed fp=0xc000051f50 sp=0xc000051e20 pc=0xc2b10d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7245
runtime.main()
        /usr/local/go/src/runtime/proc.go:254 +0x325 fp=0xc000051fe0 sp=0xc000051f50 pc=0xc1d1a5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0xc571e1

goroutine 2 gp=0xc000068700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006bfa8 sp=0xc00006bf88 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:337 +0xb8 fp=0xc00006bfe0 sp=0xc00006bfa8 pc=0xc1d418
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xc571e1
created by runtime.init.7 in goroutine 1
        /usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000068a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006df80 sp=0xc00006df60 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000078000)
        /usr/local/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006dfc8 sp=0xc00006df80 pc=0xc06334
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0xbfa965
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000068c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000078000?, 0x1c7a718?, 0x1?, 0x0?, 0xc000068c40?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x25264c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0xc03d69
runtime.bgscavenge(0xc000078000)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000085fc8 sp=0xc000085fa8 pc=0xc042dc
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0xbfa905
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000068fc0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000087e20 sp=0xc000087e00 pc=0xc4eb8e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000087fe0 sp=0xc000087e20 pc=0xbf9a27
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0xc571e1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:163 +0x3d
PS C:\restic> .\restic_0.17.0_windows_amd64_g123.exe version
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1acd5ed?, 0x6?})
        /usr/local/go/src/runtime/panic.go:1067 +0x4d fp=0xc00048fb78 sp=0xc00048fb48 pc=0xc4ea6d
runtime.badmcall(0x8?)
        /usr/local/go/src/runtime/proc.go:539 +0x1f fp=0xc00048fb98 sp=0xc00048fb78 pc=0xc1daff
runtime.badmcall(0xc1bc99)
        <autogenerated>:1 +0x25 fp=0xc00048fbb0 sp=0xc00048fb98 pc=0xc5bf25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00048fbc8
stack: frame={sp:0xc00048fb98, fp:0xc00048fbb0} stack=[0x4eb6604000,0x4eb67ffc10)


goroutine 1 gp=0xc000068000 m=nil [runnable, locked to thread]:
github.com/restic/restic/internal/restic.SnapshotGroupByOptions.String({0x0?, 0x1?, 0x1?})
        /restic/internal/restic/snapshot_group.go:34 +0x17c fp=0xc00048fd68 sp=0xc00048fd60 pc=0x106a21c
github.com/restic/restic/internal/restic.(*SnapshotGroupByOptions).String(0xc0004e0100?)
        <autogenerated>:1 +0x2e fp=0xc00048fd80 sp=0xc00048fd68 pc=0x1073b6e
github.com/spf13/pflag.(*FlagSet).VarPF(0xc0004e0100, {0x1c90130, 0x25273d0}, {0x1aa01de, 0x8}, {0x1c7a740, 0x1}, {0x1af4025, 0x5b})
        /home/build/go/pkg/mod/github.com/spf13/[email protected]/flag.go:829 +0x48 fp=0xc00048fdb8 sp=0xc00048fd80 pc=0xf5c028
github.com/spf13/pflag.(*FlagSet).VarP(...)
        /home/build/go/pkg/mod/github.com/spf13/[email protected]/flag.go:837
main.init.0()
        /restic/cmd/restic/cmd_backup.go:106 +0x108 fp=0xc00048fe20 sp=0xc00048fdb8 pc=0x172d2a8
runtime.doInit1(0x248c800)
        /usr/local/go/src/runtime/proc.go:7278 +0xed fp=0xc00048ff50 sp=0xc00048fe20 pc=0xc2b10d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7245
runtime.main()
        /usr/local/go/src/runtime/proc.go:254 +0x325 fp=0xc00048ffe0 sp=0xc00048ff50 pc=0xc1d1a5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00048ffe8 sp=0xc00048ffe0 pc=0xc571e1

goroutine 2 gp=0xc000068700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006bfa8 sp=0xc00006bf88 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:337 +0xb8 fp=0xc00006bfe0 sp=0xc00006bfa8 pc=0xc1d418
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xc571e1
created by runtime.init.7 in goroutine 1
        /usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000068a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006df80 sp=0xc00006df60 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000078000)
        /usr/local/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006dfc8 sp=0xc00006df80 pc=0xc06334
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0xbfa965
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000068c40 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x1c7a718?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x25264c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0xc03d69
runtime.bgscavenge(0xc000078000)
        /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000085fc8 sp=0xc000085fa8 pc=0xc042f9
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0xbfa905
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000068fc0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000087e20 sp=0xc000087e00 pc=0xc4eb8e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000087fe0 sp=0xc000087e20 pc=0xbf9a27
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0xc571e1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:163 +0x3d

goroutine 6 gp=0xc0001ab6c0 m=nil [select]:
runtime.gopark(0xc00006ff78?, 0x3?, 0x90?, 0x0?, 0xc00006ff72?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006fe10 sp=0xc00006fdf0 pc=0xc4eb8e
runtime.selectgo(0xc00006ff78, 0xc00006ff6c, 0xc00015b700?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:335 +0x7a5 fp=0xc00006ff38 sp=0xc00006fe10 pc=0xc2df05
go.opencensus.io/stats/view.(*worker).start(0xc00015b700)
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f fp=0xc00006ffc8 sp=0xc00006ff38 pc=0x16332df
go.opencensus.io/stats/view.init.0.gowrap1()
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x25 fp=0xc00006ffe0 sp=0xc00006ffc8 pc=0x1632645
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0xc571e1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        /home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 18 gp=0xc00008e380 m=nil [chan receive]:
runtime.gopark(0xc000081f60?, 0xdbfd05?, 0x30?, 0x23?, 0x1ca8ac0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000081f18 sp=0xc000081ef8 pc=0xc4eb8e
runtime.chanrecv(0xc0000c0000, 0x0, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41e fp=0xc000081f90 sp=0xc000081f18 pc=0xbea41e
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:489 +0x12 fp=0xc000081fb8 sp=0xc000081f90 pc=0xbe9fd2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        /usr/local/go/src/runtime/mgc.go:1732
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1735 +0x2f fp=0xc000081fe0 sp=0xc000081fb8 pc=0xbfda0f
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000081fe8 sp=0xc000081fe0 pc=0xc571e1
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1730 +0x96

goroutine 34 gp=0xc00035c000 m=nil [chan receive]:
runtime.gopark(0xcbef05?, 0x18ee5c0?, 0x1?, 0x1a?, 0xc000363f60?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000363ed0 sp=0xc000363eb0 pc=0xc4eb8e
runtime.chanrecv(0xc0003b29a0, 0xc000363fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41e fp=0xc000363f48 sp=0xc000363ed0 pc=0xbea41e
runtime.chanrecv2(0xc0003b29a0?, 0xc000363fc0?)
        /usr/local/go/src/runtime/chan.go:494 +0x12 fp=0xc000363f70 sp=0xc000363f48 pc=0xbe9ff2
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:404 +0x94 fp=0xc000363fe0 sp=0xc000363f70 pc=0x1070834
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000363fe8 sp=0xc000363fe0 pc=0xc571e1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
        /restic/internal/restic/lock.go:401 +0x1a

goroutine 19 gp=0xc00008e700 m=3 mp=0xc000075008 [syscall]:
runtime.notetsleepg(0x25701c0, 0xffffffffffffffff)
        /usr/local/go/src/runtime/lock_sema.go:296 +0x31 fp=0xc000083fa0 sp=0xc000083f68 pc=0xbefd51
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000083fc0 sp=0xc000083fa0 pc=0xc502e9
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000083fe0 sp=0xc000083fc0 pc=0xcbf2d3
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000083fe8 sp=0xc000083fe0 pc=0xc571e1
created by os/signal.Notify.func1.1 in goroutine 34
        /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 35 gp=0xc00035cc40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000365f38 sp=0xc000365f18 pc=0xc4eb8e
runtime.gcBgMarkWorker(0xc000344150)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc000365fc8 sp=0xc000365f38 pc=0xbfcd09
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc000365fe0 sp=0xc000365fc8 pc=0xbfcbe5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000365fe8 sp=0xc000365fe0 pc=0xc571e1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 7 gp=0xc0001aba40 m=8 mp=0xc00044c008 [GC mark termination]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:479 +0x8 fp=0xc00035fe98 sp=0xc00035fe88 pc=0xc551c8
runtime.forEachP(...)
        /usr/local/go/src/runtime/proc.go:2000
runtime.gcMarkDone()
        /usr/local/go/src/runtime/mgc.go:847 +0x152 fp=0xc00035ff38 sp=0xc00035fe98 pc=0xbfb5d2
runtime.gcBgMarkWorker(0xc000344150)
        /usr/local/go/src/runtime/mgc.go:1499 +0x345 fp=0xc00035ffc8 sp=0xc00035ff38 pc=0xbfcf65
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00035ffe0 sp=0xc00035ffc8 pc=0xbfcbe5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00035ffe8 sp=0xc00035ffe0 pc=0xc571e1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 8 gp=0xc0001abc00 m=nil [GC worker (idle)]:
runtime.gopark(0xa4ad77be4?, 0x3?, 0x18?, 0xa7?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000361f38 sp=0xc000361f18 pc=0xc4eb8e
runtime.gcBgMarkWorker(0xc000344150)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc000361fc8 sp=0xc000361f38 pc=0xbfcd09
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc000361fe0 sp=0xc000361fc8 pc=0xbfcbe5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000361fe8 sp=0xc000361fe0 pc=0xc571e1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 20 gp=0xc00008e8c0 m=nil [semacquire]:
runtime.gopark(0xc0004b8000?, 0xc55217?, 0x0?, 0xc0?, 0xbe9889?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0003e5e30 sp=0xc0003e5e10 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.semacquire1(0x2527efc, 0x0, 0x0, 0x0, 0x12)
        /usr/local/go/src/runtime/sema.go:178 +0x232 fp=0xc0003e5e98 sp=0xc0003e5e30 pc=0xc2efd2
runtime.semacquire(...)
        /usr/local/go/src/runtime/sema.go:129
runtime.gcMarkDone()
        /usr/local/go/src/runtime/mgc.go:827 +0x32 fp=0xc0003e5f38 sp=0xc0003e5e98 pc=0xbfb4b2
runtime.gcBgMarkWorker(0xc000344150)
        /usr/local/go/src/runtime/mgc.go:1499 +0x345 fp=0xc0003e5fc8 sp=0xc0003e5f38 pc=0xbfcf65
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0003e5fe0 sp=0xc0003e5fc8 pc=0xbfcbe5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0003e5fe8 sp=0xc0003e5fe0 pc=0xc571e1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105
PS C:\restic> .\restic_0.17.0_windows_amd64_g123.exe version
restic 0.17.0 compiled with go1.23.0 on windows/amd64
PS C:\restic>

Once the app runs successfully, it seems to be relatively stable for successive executions, but it will eventually panic again.

Aug 21 '24 19:08 distancesprinter

A few more data points:

I compiled restic with every point version of go between 1.20.3 and 1.23.0.
I wrote a PowerShell script to execute restic with the version option 20 times.
I found that restic compiled with versions of go 1.21.4 and earlier did not panic.
The version of restic compiled with go version 1.21.5 panicked during the first 3 of 20 iterations.
All future version of go panicked, but I am not sharing output of 1.21.6+ for brevity, except for 1.22.5 (because this is the version for the release build of restic) and 1.23.0 (to show the problem is present in the latest release).
The version of restic compiled with go version 1.22.5 (the version of go used by restic release build) panicked during the first 4 of 20 iterations.
The version of restic compiled with go version 1.23.0 panicked during the first 2 of 20 iterations.
The problem did not present itself on my laptop running Windows 10 Enterprise build 19045 with the following CPU: AMD Ryzen 5 PRO 5650U with Radeon Graphics

Methodology:

Compile restic in docker container as per restic build procedure.
For each build: (1) Reboot system. (2) Execute iterations for specific restic build running PowerShell script.

PowerShell:

$ResticBin="C:\restic\restic_0.17.0_windows_amd64_g1214.exe"
$ResticCmd="version"

$N=0; $Limit=Read-Host "Please input loop limit"
While ($N -lt $Limit) { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1 }

# Run the following at PS command prompt:
# .\iter_g1214.ps1 6>&1 5>&1 4>&1 3>&1 2>&1 | tee -filePath "C:\restic\out_g1214.txt"

out_g1214.txt:

Loop 0 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 1 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 2 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 3 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 4 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 5 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 6 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 7 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 8 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 9 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 10 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 11 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 12 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 13 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 14 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 15 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 16 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 17 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 18 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 19 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64

out_g1215.txt:

Loop 0 in 20
restic_0.17.0_windows_amd64_g1215.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g1215.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 

runtime stack:
runtime.throw({0x107f27f?, 0x1a4dda0?})
	/usr/local/go/src/runtime/panic.go:1077 +0x65 fp=0xc000145c20 sp=0xc000145bf0 pc=0x23b1c5
runtime.badmcall(0xc000145c78?)
	/usr/local/go/src/runtime/proc.go:503 +0x1f fp=0xc000145c40 sp=0xc000145c20 pc=0x23e4bf
runtime.badmcall(0x23c539)
	<autogenerated>:1 +0x25 fp=0xc000145c58 sp=0xc000145c40 pc=0x2706c5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000145c70
stack: frame={sp:0xc000145c40, fp:0xc000145c58} stack=[0x3d41804000,0x3d419ffc70)


goroutine 1 [runnable, locked to thread]:
github.com/Azure/azure-sdk-for-go/sdk/internal/log.init.0()
	/home/build/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/log/log.go:90 +0x16 fp=0xc000145e10 
sp=0xc000145e08 pc=0x88f716
runtime.doInit1(0x19e51e0)
	/usr/local/go/src/runtime/proc.go:6740 +0xdd fp=0xc000145f40 sp=0xc000145e10 pc=0x24a93d
runtime.doInit(...)
	/usr/local/go/src/runtime/proc.go:6707
runtime.main()
	/usr/local/go/src/runtime/proc.go:249 +0x356 fp=0xc000145fe0 sp=0xc000145f40 pc=0x23dab6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000145fe8 sp=0xc000145fe0 pc=0x26ba61

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000061fa8 sp=0xc000061f88 pc=0x23de2e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:322 +0xb8 fp=0xc000061fe0 sp=0xc000061fa8 pc=0x23dcb8
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x26ba61
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 18 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00010df78 sp=0xc00010df58 pc=0x23de2e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:280 +0x94 fp=0xc00010dfc8 sp=0xc00010df78 pc=0x2283f4
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc00010dfe0 sp=0xc00010dfc8 pc=0x21d4e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 19 [GC scavenge wait]:
runtime.gopark(0xc00006c000?, 0x1223630?, 0x1?, 0x0?, 0xc000104340?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00010ff70 sp=0xc00010ff50 pc=0x23de2e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1a7c5a0)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00010ffa0 sp=0xc00010ff70 pc=0x225cc9
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00010ffc8 sp=0xc00010ffa0 pc=0x22625c
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x21d485
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 20 [finalizer wait]:
runtime.gopark(0x1876b1b0b30?, 0x400000?, 0x70?, 0x5e?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000065e20 sp=0xc000065e00 pc=0x23de2e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000065fe0 sp=0xc000065e20 pc=0x21c547
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x26ba61
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:163 +0x3d
Loop 1 in 20
restic_0.17.0_windows_amd64_g1215.exe : fatal: morestack on g0
At C:\restic\iter_g1215.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal: morestack on g0:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
Exception 0x80000003 0x0 0x0 0x26b8c1
PC=0x26b8c1

runtime.abort()
	/usr/local/go/src/runtime/asm_amd64.s:1113 +0x1 fp=0xc0001197d0 sp=0xc0001197c8 pc=0x26b8c1
traceback: unexpected SPWRITE function runtime.morestack
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:560 +0x24 fp=0xc0001197d8 sp=0xc0001197d0 pc=0x269b24

goroutine 1 [runnable, locked to thread]:
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc000008510, {0x1244428?, 0xc00015a000?})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/reflect/protoregistry/registry.go:114 +0xb86 
fp=0xc0001199b0 sp=0xc0001199a8 pc=0xa0c466
google.golang.org/protobuf/internal/filetype.(*resolverByIndex).RegisterFile(0xddf5c1?, {0x1244428?, 0xc00015a000?})
	<autogenerated>:1 +0x2f fp=0xc0001199d8 sp=0xc0001199b0 pc=0xa9d18f
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0xddf5c1, 0x37}, {0x19f18c0, 0x226, 0x226}, 0x2, 0x2, 
0x0, 0x0, {0x12353d0, ...}, ...})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/internal/filedesc/build.go:112 +0x1d6 fp=0xc000119a40 
sp=0xc0001199d8 pc=0xa28e56
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0xddf5c1, 0x37}, {0x19f18c0, 0x226, 0x226}, 0x2, 0x2, 
0x0, 0x0, {0x0, ...}, ...}, ...})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/internal/filetype/build.go:138 +0x199 fp=0xc000119c20 
sp=0xc000119a40 pc=0xa9ae79
github.com/google/s2a-go/internal/proto/common_go_proto.file_internal_proto_common_common_proto_init()
	/home/build/go/pkg/mod/github.com/google/[email protected]/internal/proto/common_go_proto/common.pb.go:384 +0x278 
fp=0xc000119e00 sp=0xc000119c20 pc=0xb6ff38
github.com/google/s2a-go/internal/proto/common_go_proto.init.0()
	/home/build/go/pkg/mod/github.com/google/[email protected]/internal/proto/common_go_proto/common.pb.go:344 +0xf 
fp=0xc000119e10 sp=0xc000119e00 pc=0xb6fcaf
runtime.doInit1(0x19e6860)
	/usr/local/go/src/runtime/proc.go:6740 +0xdd fp=0xc000119f40 sp=0xc000119e10 pc=0x24a93d
runtime.doInit(...)
	/usr/local/go/src/runtime/proc.go:6707
runtime.main()
	/usr/local/go/src/runtime/proc.go:249 +0x356 fp=0xc000119fe0 sp=0xc000119f40 pc=0x23dab6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0x26ba61

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000061fa8 sp=0xc000061f88 pc=0x23de2e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:322 +0xb8 fp=0xc000061fe0 sp=0xc000061fa8 pc=0x23dcb8
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x26ba61
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000063f78 sp=0xc000063f58 pc=0x23de2e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:280 +0x94 fp=0xc000063fc8 sp=0xc000063f78 pc=0x2283f4
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x21d4e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00006c000?, 0x1223630?, 0x1?, 0x0?, 0xc00005eb60?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000073f70 sp=0xc000073f50 pc=0x23de2e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1a7c5a0)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000073fa0 sp=0xc000073f70 pc=0x225cc9
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000073fc8 sp=0xc000073fa0 pc=0x22625c
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000073fe0 sp=0xc000073fc8 pc=0x21d485
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 5 [finalizer wait]:
runtime.gopark(0x2100be?, 0x400000?, 0x70?, 0x5e?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000065e20 sp=0xc000065e00 pc=0x23de2e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000065fe0 sp=0xc000065e20 pc=0x21c547
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x26ba61
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:163 +0x3d
rax     0x17
rbx     0x1a7dc40
rcx     0x1a7d9c0
rdi     0xc8c1a7f000
rsi     0xc0001196c8
rbp     0xc0001197e8
rsp     0xc0001197c8
r8      0xc000119498
r9      0xc000119670
r10     0x0
r11     0x246
r12     0xc8c1dff6a0
r13     0x1
r14     0x1a7d3a0
r15     0x19f18ef
rip     0x26b8c1
rflags  0x216
cs      0x33
fs      0x53
gs      0x2b
Loop 2 in 20
restic_0.17.0_windows_amd64_g1215.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g1215.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 

runtime stack:
runtime.throw({0x107f27f?, 0x218948?})
	/usr/local/go/src/runtime/panic.go:1077 +0x65 fp=0xc000099c20 sp=0xc000099bf0 pc=0x23b1c5
runtime.badmcall(0x1ac8a503d50?)
	/usr/local/go/src/runtime/proc.go:503 +0x1f fp=0xc000099c40 sp=0xc000099c20 pc=0x23e4bf
runtime.badmcall(0x23c539)
	<autogenerated>:1 +0x25 fp=0xc000099c58 sp=0xc000099c40 pc=0x2706c5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000099c70
stack: frame={sp:0xc000099c40, fp:0xc000099c58} stack=[0x60d5204000,0x60d53ff900)


goroutine 1 [runnable, locked to thread]:
github.com/restic/restic/internal/filter.init()
	/restic/internal/filter/filter.go:12 +0x5b fp=0xc000099e10 sp=0xc000099e08 pc=0x81361b
runtime.doInit1(0x19e4c40)
	/usr/local/go/src/runtime/proc.go:6740 +0xdd fp=0xc000099f40 sp=0xc000099e10 pc=0x24a93d
runtime.doInit(...)
	/usr/local/go/src/runtime/proc.go:6707
runtime.main()
	/usr/local/go/src/runtime/proc.go:249 +0x356 fp=0xc000099fe0 sp=0xc000099f40 pc=0x23dab6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000099fe8 sp=0xc000099fe0 pc=0x26ba61

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000061fa8 sp=0xc000061f88 pc=0x23de2e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:322 +0xb8 fp=0xc000061fe0 sp=0xc000061fa8 pc=0x23dcb8
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x26ba61
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000063f78 sp=0xc000063f58 pc=0x23de2e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:280 +0x94 fp=0xc000063fc8 sp=0xc000063f78 pc=0x2283f4
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x21d4e5
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00006c000?, 0x1223630?, 0x1?, 0x0?, 0xc00005eb60?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000073f70 sp=0xc000073f50 pc=0x23de2e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1a7c5a0)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000073fa0 sp=0xc000073f70 pc=0x225cc9
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000073fc8 sp=0xc000073fa0 pc=0x22625c
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000073fe0 sp=0xc000073fc8 pc=0x21d485
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 5 [finalizer wait]:
runtime.gopark(0x2100be?, 0x400000?, 0x70?, 0x5e?, 0x0?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000065e20 sp=0xc000065e00 pc=0x23de2e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000065fe0 sp=0xc000065e20 pc=0x21c547
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x26ba61
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:163 +0x3d
Loop 3 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 4 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 5 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 6 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 7 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 8 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 9 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 10 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 11 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 12 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 13 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 14 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 15 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 16 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 17 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 18 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 19 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64

out_g1225.txt:

Loop 0 in 20
restic_0.17.0_windows_amd64.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g1225.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 

runtime stack:
runtime.throw({0x1c72ad1?, 0x25?})
	/usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc0000a3c30 sp=0xc0000a3c00 pc=0xe0d405
runtime.badmcall(0xc0000a3ca8?)
	/usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc0000a3c50 sp=0xc0000a3c30 pc=0xe107bf
runtime.badmcall(0xe0e899)
	<autogenerated>:1 +0x25 fp=0xc0000a3c68 sp=0xc0000a3c50 pc=0xe45be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc0000a3c80
stack: frame={sp:0xc0000a3c50, fp:0xc0000a3c68} stack=[0xac86604000,0xac867ff710)


goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
compress/flate.init()
	<autogenerated>:1 +0xf8 fp=0xc0000a3e20 sp=0xc0000a3e18 pc=0x106fd58
runtime.doInit1(0x25f33b0)
	/usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc0000a3f50 sp=0xc0000a3e20 pc=0xe1e06d
runtime.doInit(...)
	/usr/local/go/src/runtime/proc.go:7143
runtime.main()
	/usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc0000a3fe0 sp=0xc0000a3f50 pc=0xe0fdc7
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000a3fe8 sp=0xc0000a3fe0 pc=0xe40ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xe0ffb8
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xe40ea1
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
	/usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xdfa2d4
runtime.gcenable.gowrap1()
	/usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xdeeb45
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x1e1dd08?, 0x1?, 0x0?, 0xc000066c40?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x26943c0)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0xdf7cc9
runtime.bgscavenge(0xc000074000)
	/usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0xdf825c
runtime.gcenable.gowrap2()
	/usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0xdeeae5
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0xdf70fc?, 0xc0?, 0xe5?, 0x550011?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0xe1012e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0xdedbc7
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xe40ea1
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:164 +0x3d
Loop 1 in 20
restic_0.17.0_windows_amd64.exe : fatal: morestack on g0
At C:\restic\iter_g1225.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal: morestack on g0:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
Exception 0x80000003 0x0 0x0 0xe40d01
PC=0xe40d01

runtime.abort()
	/usr/local/go/src/runtime/asm_amd64.s:1157 +0x1 fp=0xc00035fb08 sp=0xc00035fb00 pc=0xe40d01
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:593 +0x3d fp=0xc00035fb10 sp=0xc00035fb08 pc=0xe3ef7d

goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
net/http.(*Transport).RegisterProtocol(0xc0003f3180, {0x1c40a0f, 0x5}, {0x1e286e0, 0xc000426b40})
	/usr/local/go/src/net/http/transport.go:764 +0x312 fp=0xc00035fce8 sp=0xc00035fce0 pc=0x10f1d32
golang.org/x/net/http2.registerHTTPSProtocol(0xc000382000?, {0xc0002066c0?})
	/home/build/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:3219 +0x6a fp=0xc00035fd50 sp=0xc00035fce8 
pc=0x1207baa
golang.org/x/net/http2.configureTransports(0xc0003f3180)
	/home/build/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:287 +0xa5 fp=0xc00035fde8 sp=0xc00035fd50 
pc=0x11f7fe5
golang.org/x/net/http2.ConfigureTransports(...)
	/home/build/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:277
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.init.0()
	/home/build/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/runtime/transport_default_http_client.go:
39 +0x15c fp=0xc00035fe20 sp=0xc00035fde8 pc=0x14904dc
runtime.doInit1(0x25f3410)
	/usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc00035ff50 sp=0xc00035fe20 pc=0xe1e06d
runtime.doInit(...)
	/usr/local/go/src/runtime/proc.go:7143
runtime.main()
	/usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc00035ffe0 sp=0xc00035ff50 pc=0xe0fdc7
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00035ffe8 sp=0xc00035ffe0 pc=0xe40ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xe0ffb8
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xe40ea1
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
	/usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xdfa2d4
runtime.gcenable.gowrap1()
	/usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xdeeb45
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x1e1dd08?, 0x1?, 0x0?, 0xc000066c40?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x26943c0)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0xdf7cc9
runtime.bgscavenge(0xc000074000)
	/usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0xdf825c
runtime.gcenable.gowrap2()
	/usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0xdeeae5
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0xdf70fc?, 0xc0?, 0xe5?, 0x550011?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0xe1012e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0xdedbc7
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xe40ea1
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 18 gp=0xc00038a1c0 m=nil [select]:
runtime.gopark(0xc000077f78?, 0x3?, 0x98?, 0x5?, 0xc000077f72?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077e18 sp=0xc000077df8 pc=0xe1012e
runtime.selectgo(0xc000077f78, 0xc000077f6c, 0xc00009ca00?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000077f38 sp=0xc000077e18 pc=0xe20585
go.opencensus.io/stats/view.(*worker).start(0xc00009ca00)
	/home/build/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f fp=0xc000077fc8 sp=0xc000077f38 
pc=0x17e4dff
go.opencensus.io/stats/view.init.0.gowrap1()
	/home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x25 fp=0xc000077fe0 sp=0xc000077fc8 
pc=0x17e4165
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0xe40ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d
rax     0x17
rbx     0x2696378
rcx     0x2695fc0
rdx     0x0
rdi     0xf850475000
rsi     0xc00035fa00
rbp     0xc00035fb20
rsp     0xc00035fb00
r8      0xc00035f7d8
r9      0xc00035f9b0
r10     0x0
r11     0x246
r12     0xf8507ffd30
r13     0x0
r14     0x26953c0
r15     0x1fffffffffff
rip     0xe40d01
rflags  0x202
cs      0x33
fs      0x53
gs      0x2b
Loop 2 in 20
restic_0.17.0_windows_amd64.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g1225.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 

runtime stack:
runtime.throw({0x1c72ad1?, 0xcb?})
	/usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc00035fc30 sp=0xc00035fc00 pc=0xe0d405
runtime.badmcall(0xc00035fc98?)
	/usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc00035fc50 sp=0xc00035fc30 pc=0xe107bf
runtime.badmcall(0xe0e899)
	<autogenerated>:1 +0x25 fp=0xc00035fc68 sp=0xc00035fc50 pc=0xe45be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00035fc80
stack: frame={sp:0xc00035fc50, fp:0xc00035fc68} stack=[0x7d9c604000,0x7d9c7ffcd0)


goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
main.init.36()
	/restic/cmd/restic/main.go:23 +0x26 fp=0xc00053fe20 sp=0xc00053fe18 pc=0x191a126
runtime.doInit1(0x25fb3e0)
	/usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc00053ff50 sp=0xc00053fe20 pc=0xe1e06d
runtime.doInit(...)
	/usr/local/go/src/runtime/proc.go:7143
runtime.main()
	/usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc00053ffe0 sp=0xc00053ff50 pc=0xe0fdc7
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0xe40ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xe0ffb8
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xe40ea1
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
	/usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xdfa2d4
runtime.gcenable.gowrap1()
	/usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xdeeb45
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x1e1dd08?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x26943c0)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0xdf7cc9
runtime.bgscavenge(0xc000074000)
	/usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0xdf8279
runtime.gcenable.gowrap2()
	/usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0xdeeae5
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0xdf70fc?, 0xc0?, 0xe5?, 0x550011?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0xe1012e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0xdedbc7
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xe40ea1
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 6 gp=0xc0001b76c0 m=nil [select]:
runtime.gopark(0xc00007df78?, 0x3?, 0x98?, 0x5?, 0xc00007df72?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007de18 sp=0xc00007ddf8 pc=0xe1012e
runtime.selectgo(0xc00007df78, 0xc00007df6c, 0xc0000e5700?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00007df38 sp=0xc00007de18 pc=0xe20585
go.opencensus.io/stats/view.(*worker).start(0xc0000e5700)
	/home/build/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f fp=0xc00007dfc8 sp=0xc00007df38 
pc=0x17e4dff
go.opencensus.io/stats/view.init.0.gowrap1()
	/home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 
pc=0x17e4165
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0xe40ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 7 gp=0xc0001b7880 m=nil [chan receive]:
runtime.gopark(0xea5c05?, 0x1a9a9c0?, 0x1?, 0x80?, 0xc000077f60?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077ed0 sp=0xc000077eb0 pc=0xe1012e
runtime.chanrecv(0xc00010c120, 0xc000077fa0, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000077f48 sp=0xc000077ed0 pc=0xddaa0d
runtime.chanrecv2(0xc00010c120?, 0xc000077fc0?)
	/usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc000077f70 sp=0xc000077f48 pc=0xdda632
github.com/restic/restic/internal/restic.init.0.func1.1()
	/restic/internal/restic/lock.go:404 +0x99 fp=0xc000077fe0 sp=0xc000077f70 pc=0x12292f9
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0xe40ea1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
	/restic/internal/restic/lock.go:401 +0x1a

goroutine 18 gp=0xc0001041c0 m=4 mp=0xc000073808 [syscall]:
runtime.notetsleepg(0x271d9a0, 0xffffffffffffffff)
	/usr/local/go/src/runtime/lock_sema.go:296 +0x31 fp=0xc000117fa0 sp=0xc000117f68 pc=0xde0771
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000117fc0 sp=0xc000117fa0 pc=0xe3c789
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000117fe0 sp=0xc000117fc0 pc=0xea5fd3
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0xe40ea1
created by os/signal.Notify.func1.1 in goroutine 7
	/usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 34 gp=0xc000405c00 m=nil [GC worker (idle)]:
runtime.gopark(0x271e5c0?, 0x3?, 0xd4?, 0xf4?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000113f50 sp=0xc000113f30 pc=0xe1012e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000113fe0 sp=0xc000113f50 pc=0xdf0c85
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000113fe8 sp=0xc000113fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 19 gp=0xc000104380 m=nil [GC worker (idle)]:
runtime.gopark(0x271e5c0?, 0x3?, 0xb8?, 0x78?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000119f50 sp=0xc000119f30 pc=0xe1012e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000119fe0 sp=0xc000119f50 pc=0xdf0c85
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 8 gp=0xc0001b7c00 m=nil [GC worker (idle)]:
runtime.gopark(0x271e5c0?, 0x3?, 0xb8?, 0x78?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000079f50 sp=0xc000079f30 pc=0xe1012e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000079fe0 sp=0xc000079f50 pc=0xdf0c85
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 35 gp=0xc000405dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x271e5c0?, 0x1?, 0x14?, 0x23?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000115f50 sp=0xc000115f30 pc=0xe1012e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000115fe0 sp=0xc000115f50 pc=0xdf0c85
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000115fe8 sp=0xc000115fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c
Loop 3 in 20
restic_0.17.0_windows_amd64.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g1225.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 

runtime stack:
runtime.throw({0x1c72ad1?, 0x374000?})
	/usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc00053fc10 sp=0xc00053fbe0 pc=0xe0d405
runtime.badmcall(0x10?)
	/usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc00053fc30 sp=0xc00053fc10 pc=0xe107bf
runtime.badmcall(0xe0e899)
	<autogenerated>:1 +0x25 fp=0xc00053fc48 sp=0xc00053fc30 pc=0xe45be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00053fc60
stack: frame={sp:0xc00053fc30, fp:0xc00053fc48} stack=[0xbb31204000,0xbb313ffbe0)


goroutine 1 gp=0xc000066000 m=nil [runnable]:
github.com/spf13/cobra.(*Command).ExecuteC(0x26845e0)
	/home/build/go/pkg/mod/github.com/spf13/[email protected]/command.go:1054 +0x625 fp=0xc00053fe00 sp=0xc00053fdf8 
pc=0x1178b05
github.com/spf13/cobra.(*Command).Execute(...)
	/home/build/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/build/go/pkg/mod/github.com/spf13/[email protected]/command.go:1034
main.main()
	/restic/cmd/restic/main.go:125 +0x265 fp=0xc00053ff50 sp=0xc00053fe00 pc=0x191a3e5
runtime.main()
	/usr/local/go/src/runtime/proc.go:271 +0x28b fp=0xc00053ffe0 sp=0xc00053ff50 pc=0xe0fd2b
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0xe40ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xe0ffb8
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xe40ea1
created by runtime.init.6 in goroutine 1
	/usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
	/usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xdfa2d4
runtime.gcenable.gowrap1()
	/usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xdeeb45
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x1e1dd08?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0xe1012e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x26943c0)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0xdf7cc9
runtime.bgscavenge(0xc000074000)
	/usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0xdf8279
runtime.gcenable.gowrap2()
	/usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0xdeeae5
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0xdf70fc?, 0xc0?, 0xe5?, 0x550011?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0xe1012e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0xdedbc7
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xe40ea1
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 6 gp=0xc0001376c0 m=nil [select]:
runtime.gopark(0xc00007df78?, 0x3?, 0x28?, 0xa?, 0xc00007df72?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007de18 sp=0xc00007ddf8 pc=0xe1012e
runtime.selectgo(0xc00007df78, 0xc00007df6c, 0xc0000e5700?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00007df38 sp=0xc00007de18 pc=0xe20585
go.opencensus.io/stats/view.(*worker).start(0xc0000e5700)
	/home/build/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f fp=0xc00007dfc8 sp=0xc00007df38 
pc=0x17e4dff
go.opencensus.io/stats/view.init.0.gowrap1()
	/home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 
pc=0x17e4165
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0xe40ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/home/build/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 7 gp=0xc000137880 m=nil [chan receive]:
runtime.gopark(0xea5c05?, 0x1a9a9c0?, 0x1?, 0x20?, 0xc000077f60?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077ed0 sp=0xc000077eb0 pc=0xe1012e
runtime.chanrecv(0xc0002e6120, 0xc000077fa0, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000077f48 sp=0xc000077ed0 pc=0xddaa0d
runtime.chanrecv2(0xc0002e6120?, 0xc000077fc0?)
	/usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc000077f70 sp=0xc000077f48 pc=0xdda632
github.com/restic/restic/internal/restic.init.0.func1.1()
	/restic/internal/restic/lock.go:404 +0x99 fp=0xc000077fe0 sp=0xc000077f70 pc=0x12292f9
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0xe40ea1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
	/restic/internal/restic/lock.go:401 +0x1a

goroutine 18 gp=0xc0002ec1c0 m=4 mp=0xc000073808 [syscall]:
runtime.notetsleepg(0x271d9a0, 0xffffffffffffffff)
	/usr/local/go/src/runtime/lock_sema.go:296 +0x31 fp=0xc0002f7fa0 sp=0xc0002f7f68 pc=0xde0771
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc0002f7fc0 sp=0xc0002f7fa0 pc=0xe3c789
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0002f7fe0 sp=0xc0002f7fc0 pc=0xea5fd3
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f7fe8 sp=0xc0002f7fe0 pc=0xe40ea1
created by os/signal.Notify.func1.1 in goroutine 7
	/usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 8 gp=0xc000391500 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000079f50 sp=0xc000079f30 pc=0xe1012e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000079fe0 sp=0xc000079f50 pc=0xdf0c85
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 34 gp=0xc0004841c0 m=nil [GC worker (idle)]:
runtime.gopark(0x11b72e1120?, 0x3?, 0xbc?, 0x5c?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0002f3f50 sp=0xc0002f3f30 pc=0xe1012e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0002f3fe0 sp=0xc0002f3f50 pc=0xdf0c85
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f3fe8 sp=0xc0002f3fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 9 gp=0xc000391880 m=0 mp=0x2695fc0 [GC mark termination]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc00040fec0 sp=0xc00040feb0 pc=0xe3ee88
runtime.forEachP(...)
	/usr/local/go/src/runtime/proc.go:1895
runtime.gcMarkDone()
	/usr/local/go/src/runtime/mgc.go:827 +0x152 fp=0xc00040ff50 sp=0xc00040fec0 pc=0xdef752
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1446 +0x347 fp=0xc00040ffe0 sp=0xc00040ff50 pc=0xdf0ee7
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00040ffe8 sp=0xc00040ffe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 19 gp=0xc0002ec380 m=nil [GC worker (idle)]:
runtime.gopark(0x11b72e1120?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0002f9f50 sp=0xc0002f9f30 pc=0xe1012e
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0002f9fe0 sp=0xc0002f9f50 pc=0xdf0c85
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f9fe8 sp=0xc0002f9fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 35 gp=0xc000484380 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0002f5e98 sp=0xc0002f5e78 pc=0xe1012e
runtime.chanrecv(0xc000492000, 0xc0002f5fa0, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc0002f5f10 sp=0xc0002f5e98 pc=0xddaa0d
runtime.chanrecv1(0x0?, 0x0?)
	/usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc0002f5f38 sp=0xc0002f5f10 pc=0xdda612
main.cleanupHandler(0x0?, 0xc000490000)
	/restic/cmd/restic/cleanup.go:24 +0x2a fp=0xc0002f5fc0 sp=0xc0002f5f38 pc=0x18df8aa
main.createGlobalContext.gowrap1()
	/restic/cmd/restic/cleanup.go:16 +0x25 fp=0xc0002f5fe0 sp=0xc0002f5fc0 pc=0x18df845
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f5fe8 sp=0xc0002f5fe0 pc=0xe40ea1
created by main.createGlobalContext in goroutine 1
	/restic/cmd/restic/cleanup.go:16 +0x98
Loop 4 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 5 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 6 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 7 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 8 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 9 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 10 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 11 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 12 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 13 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 14 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 15 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 16 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 17 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 18 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 19 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64

out_g123:

Loop 0 in 20
restic_0.17.0_windows_amd64_g123.exe : fatal: morestack on g0
At C:\restic\iter_g123.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal: morestack on g0:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
Exception 0x80000003 0x0 0x0 0xbc7041
PC=0xbc7041

runtime.abort()
	/usr/local/go/src/runtime/asm_amd64.s:1162 +0x1 fp=0xc000051408 sp=0xc000051400 pc=0xbc7041
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:598 +0x3d fp=0xc000051410 sp=0xc000051408 pc=0xbc52bd

goroutine 1 gp=0xc000068000 m=nil [runnable, locked to thread]:
google.golang.org/protobuf/internal/impl.getExtensionFieldInfo({0x1c085b0?, 0x247c400?})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/internal/impl/codec_extension.go:24 +0xe5 fp=0xc0000515e0 
sp=0xc0000515d8 pc=0x13ff4c5
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalExtension(0xc000189860, {0x23f83a9, 0x1, 0x54}, 
0x3ea, 0x2, 0xc0001e8480, {0x0, {0x1bfe478, 0xc00009a720}, ...})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:215 +0x285 fp=0xc000051720 
sp=0xc0000515e0 pc=0x1430085
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc000189860, {0x23f83a7?, 0x3, 0x56?}, 
{0x1409c91?}, 0x0, {0x0, {0x1bfe478, 0xc00009a720}, 0x270e})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:169 +0x5da fp=0xc000051858 
sp=0xc000051720 pc=0x142f81a
google.golang.org/protobuf/internal/impl.consumeMessageInfo({0x23f83a6?, 0x23fff20?, 0xc0000519b0?}, {0xbb995d?}, 
0x0?, 0xc0000a48a8, {0x0, {0x1bfe478, 0xc00009a720}, 0x270e})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/internal/impl/codec_field.go:258 +0x11b fp=0xc0000518e0 
sp=0xc000051858 pc=0x140277b
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc00018a678, {0x23f83a2?, 0x1b, 0x5b?}, 
{0x0?}, 0x0, {0x0, {0x1bfe478, 0xc00009a720}, 0x270f})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:148 +0x3a9 fp=0xc000051a18 
sp=0xc0000518e0 pc=0x142f5e9
google.golang.org/protobuf/internal/impl.consumeMessageSliceInfo({0x23f83a1?, 0x0?, 0xc0000a4680?}, {0x0?}, 0x7?, 
0xc0000a4680, {0x0, {0x1bfe478, 0xc00009a720}, 0x270f})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/internal/impl/codec_field.go:520 +0x105 fp=0xc000051ac0 
sp=0xc000051a18 pc=0x14048c5
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc0001899a8, {0x23f83a0?, 0x5d, 
0xc00009b1a0?}, {0xc00009b1d0?}, 0x0, {0x0, {0x1bfe478, 0xc00009a720}, 0x2710})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:148 +0x3a9 fp=0xc000051bf8 
sp=0xc000051ac0 pc=0x142f5e9
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal(0xc0001899a8?, {{}, {0x1c0e8a0, 0x2495e00}, 
{0x23f83a0, 0x5d, 0x5d}, 0x0, {0x1bfe478, 0xc00009a720}, ...})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/internal/impl/decode.go:66 +0x97 fp=0xc000051c58 
sp=0xc000051bf8 pc=0x142f1f7
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal-fm({{}, {0x1c0e8a0, 0x2495e00}, {0x23f83a0, 0x5d, 
0x5d}, 0x0, {0x1bfe478, 0xc00009a720}, 0x2710})
	<autogenerated>:1 +0x9b fp=0xc000051cb8 sp=0xc000051c58 pc=0x144f09b
google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x1, 0x1, 0x0, {0x1bfe478, 0xc00009a720}, 0x2710}, 
{0x23f83a0, 0x5d, 0x5d}, ...)
	/home/build/go/pkg/mod/google.golang.org/[email protected]/proto/decode.go:107 +0x1fb fp=0xc000051d70 sp=0xc000051cb8 
pc=0x13d227b
google.golang.org/protobuf/proto.Unmarshal({0x23f83a0, 0x5d, 0x5d}, {0x1bf9d40?, 0x2495e00?})
	/home/build/go/pkg/mod/google.golang.org/[email protected]/proto/decode.go:57 +0x5d fp=0xc000051dc8 sp=0xc000051d70 
pc=0x13d1ddd
google.golang.org/protobuf/reflect/protodesc.init.0()
	/home/build/go/pkg/mod/google.golang.org/[email protected]/reflect/protodesc/editions.go:25 +0x3a fp=0xc000051e20 
sp=0xc000051dc8 pc=0x1522d5a
runtime.doInit1(0x23f4a30)
	/usr/local/go/src/runtime/proc.go:7278 +0xed fp=0xc000051f50 sp=0xc000051e20 pc=0xb9b10d
runtime.doInit(...)
	/usr/local/go/src/runtime/proc.go:7245
runtime.main()
	/usr/local/go/src/runtime/proc.go:254 +0x325 fp=0xc000051fe0 sp=0xc000051f50 pc=0xb8d1a5
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0xbc71e1

goroutine 2 gp=0xc000068700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006bfa8 sp=0xc00006bf88 pc=0xbbeb8e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:337 +0xb8 fp=0xc00006bfe0 sp=0xc00006bfa8 pc=0xb8d418
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xbc71e1
created by runtime.init.7 in goroutine 1
	/usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000068a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006df80 sp=0xc00006df60 pc=0xbbeb8e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000078000)
	/usr/local/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006dfc8 sp=0xc00006df80 pc=0xb76334
runtime.gcenable.gowrap1()
	/usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0xb6a965
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xbc71e1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000068c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000078000?, 0x1bea718?, 0x1?, 0x0?, 0xc000068c40?)
	/usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0xbbeb8e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x24964c0)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0xb73d69
runtime.bgscavenge(0xc000078000)
	/usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000085fc8 sp=0xc000085fa8 pc=0xb742dc
runtime.gcenable.gowrap2()
	/usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0xb6a905
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0xbc71e1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000068fc0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000087e20 sp=0xc000087e00 pc=0xbbeb8e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000087fe0 sp=0xc000087e20 pc=0xb69a27
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0xbc71e1
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:163 +0x3d
rax     0x17
rbx     0x2498d80
rcx     0x2498aa0
rdx     0x0
rdi     0xd9aab8c000
rsi     0x2498aa0
rbp     0xc000051420
rsp     0xc000051400
r8      0xc0000510b8
r9      0xc000051290
r10     0x0
r11     0x246
r12     0xd9aadff790
r13     0xc00009a720
r14     0x2497640
r15     0x23f83a9
rip     0xbc7041
rflags  0x202
cs      0x33
fs      0x53
gs      0x2b
Loop 1 in 20
restic_0.17.0_windows_amd64_g123.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g123.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 

runtime stack:
runtime.throw({0x1a3d5ed?, 0x0?})
	/usr/local/go/src/runtime/panic.go:1067 +0x4d fp=0xc0001b5a18 sp=0xc0001b59e8 pc=0xbbea6d
runtime.badmcall(0xf03b7f?)
	/usr/local/go/src/runtime/proc.go:539 +0x1f fp=0xc0001b5a38 sp=0xc0001b5a18 pc=0xb8daff
runtime.badmcall(0xb8bc99)
	<autogenerated>:1 +0x25 fp=0xc0001b5a50 sp=0xc0001b5a38 pc=0xbcbf25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc0001b5a68
stack: frame={sp:0xc0001b5a38, fp:0xc0001b5a50} stack=[0x7fbe604000,0x7fbe7ff7f0)


goroutine 1 gp=0xc000068000 m=nil [runnable, locked to thread]:
regexp/syntax.(*parser).op(0xc00015b080, 0x9)
	/usr/local/go/src/regexp/syntax/parse.go:393 +0xe5 fp=0xc0001b5c00 sp=0xc0001b5bf8 pc=0xf05945
regexp/syntax.parse({0x1a53de5, 0x35}, 0xd4)
	/usr/local/go/src/regexp/syntax/parse.go:987 +0x2cf fp=0xc0001b5d00 sp=0xc0001b5c00 pc=0xf0870f
regexp/syntax.Parse(...)
	/usr/local/go/src/regexp/syntax/parse.go:887
regexp.compile({0x1a53de5, 0x35}, 0x25?, 0x0)
	/usr/local/go/src/regexp/regexp.go:168 +0x30 fp=0xc0001b5d88 sp=0xc0001b5d00 pc=0xf1a5f0
regexp.Compile(...)
	/usr/local/go/src/regexp/regexp.go:131
regexp.MustCompile({0x1a53de5, 0x35})
	/usr/local/go/src/regexp/regexp.go:311 +0x2c fp=0xc0001b5e00 sp=0xc0001b5d88 pc=0xf1b14c
go.opencensus.io/resource.init()
	/home/build/go/pkg/mod/[email protected]/resource/resource.go:60 +0x1f fp=0xc0001b5e20 sp=0xc0001b5e00 
pc=0x159cd9f
runtime.doInit1(0x23f3910)
	/usr/local/go/src/runtime/proc.go:7278 +0xed fp=0xc0001b5f50 sp=0xc0001b5e20 pc=0xb9b10d
runtime.doInit(...)
	/usr/local/go/src/runtime/proc.go:7245
runtime.main()
	/usr/local/go/src/runtime/proc.go:254 +0x325 fp=0xc0001b5fe0 sp=0xc0001b5f50 pc=0xb8d1a5
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001b5fe8 sp=0xc0001b5fe0 pc=0xbc71e1

goroutine 2 gp=0xc000068700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006bfa8 sp=0xc00006bf88 pc=0xbbeb8e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:337 +0xb8 fp=0xc00006bfe0 sp=0xc00006bfa8 pc=0xb8d418
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xbc71e1
created by runtime.init.7 in goroutine 1
	/usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000068a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006df80 sp=0xc00006df60 pc=0xbbeb8e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000078000)
	/usr/local/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006dfc8 sp=0xc00006df80 pc=0xb76334
runtime.gcenable.gowrap1()
	/usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0xb6a965
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xbc71e1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000068c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000078000?, 0x1bea718?, 0x1?, 0x0?, 0xc000068c40?)
	/usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0xbbeb8e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x24964c0)
	/usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0xb73d69
runtime.bgscavenge(0xc000078000)
	/usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000085fc8 sp=0xc000085fa8 pc=0xb742dc
runtime.gcenable.gowrap2()
	/usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0xb6a905
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0xbc71e1
created by runtime.gcenable in goroutine 1
	/usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000068fc0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000087e20 sp=0xc000087e00 pc=0xbbeb8e
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000087fe0 sp=0xc000087e20 pc=0xb69a27
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0xbc71e1
created by runtime.createfing in goroutine 1
	/usr/local/go/src/runtime/mfinal.go:163 +0x3d
Loop 2 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 3 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 4 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 5 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 6 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 7 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 8 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 9 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 10 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 11 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 12 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 13 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 14 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 15 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 16 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 17 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 18 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 19 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64

Aug 22 '24 17:08 distancesprinter

Returning here to state that, while I cannot get restic 0.17.0 built with go1.21.4 to panic with my iter script, I started using that version believing it to be stable. I can now state that I have encountered a handful of panics and I do not believe 1.21.4 to be immune from whatever bug I am experiencing. Happy to share output if & when someone gets assigned. I wanted to clarify that point in case somebody starts looking at changes between 1.21.4 to 1.21.5 to try to isolate the cause.

Aug 27 '24 14:08 distancesprinter

Just wanted to update here. We're still getting this issue with our custom golang cli tool. (Unfortunately, I can't share it). We're seeing it at a rate of about 1 to 2 crashes a day on our CI servers. Our temporary workaround has been to wrap the call of our tool with a retry. It usually works on the second attempt.

Thank you @distancesprinter for finding other repros. :)

Sep 06 '24 17:09 RichieSams

@distancesprinter

I am running Windows Server 2019 Standard (Desktop Experience) in a bhyve VM.

"In a bhyve VM" stands out to me here as the most non-standard component. Have you managed to reproduce in an environment other than a bhyve VM? (Apologies if I missed it) I wonder if this could be a register save/restore bug in bhyve.

Sep 06 '24 17:09 prattmic

@RichieSams Do I understand correctly that your crashes are occurring on bare-metal AMD machines?

Sep 06 '24 17:09 prattmic

@distancesprinter

I am running Windows Server 2019 Standard (Desktop Experience) in a bhyve VM.

"In a bhyve VM" stands out to me here as the most non-standard component. Have you managed to reproduce in an environment other than a bhyve VM? (Apologies if I missed it) I wonder if this could be a register save/restore bug in bhyve.

@prattmic I was going to point the finger at bhyve too... but without good justification. It was only a hunch since some SmartOS folks were encountering this--but I stopped myself from jumping to that conclusion because @RichieSams was seeing it on the metal.

You may be interested in reviewing/engaging in the discussion there:

I will note, the issue seemed to involve AMD processors, and mostly seemed to be presenting in Windows VMs where the issue would not occur if the VM was moved to Intel-based hardware.

SmartOS community (which uses bhyve as its hypervisor): https://smartos.topicbox.com/groups/smartos-discuss/T6938eca11dfeb1e3/smartos-discuss-golang-executables-crash-in-windows-vms-on-amd

Which I revived: https://smartos.topicbox.com/groups/smartos-discuss/Tf8f450fd133908fb/smartos-discuss-golang-executables-crash-in-windows-vms-on-amd

I would think that the bhyve folks would be interested in fixing any such bug on AMD hardware, especially as Epyc becomes more common. I think Oxide is using an AMD Epyc processor and bhyve as their hypervisor. I'm sure they'd want to track this down.

All of my AMD EPYC servers are hypervisors in production. Sadly, I cannot blow them away to test on the metal. I'd like to know if others are seeing this on the more affordable Threadripper desktop chips, though. Maybe @RichieSams could provide hardware specs where this is being seen on metal...

I'd also want to know if this is seen consistently on all Epyc hardware or just some Epyc hardware. If AMD shipped bad CPUs, that would terrify me. Mine were all bought at the same time.

Sep 06 '24 18:09 distancesprinter

Indeed, the mismatch is suspicious, but there could be different issues.

e.g., it looks like you have only gotten "mcall called on m->g0 stack" and "morestack on g0", while @RichieSams has gotten a slightly wider variety of other corruption-looking crashes.

Sep 06 '24 18:09 prattmic

runtime: fatal error: mcall called on m->g0 stack on Windows

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Output of `go env` in your module/workspace: