go
go copied to clipboard
golang
runtime,cmd/compile: `unexpected fault address` in `runtime.mapassign`
#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`
Issue created automatically to collect these failures.
Example (log):
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x21 pc=0xb91b7a]
goroutine 1 [running]:
cmd/link/internal/loader.(*Loader).OuterSym(...)
C:/workdir/go/src/cmd/link/internal/loader/loader.go:1695
cmd/link/internal/ld.(*deadcodePass).flood(0xc00006aa00)
C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:239 +0x265
cmd/link/internal/ld.deadcode(0xc0000c0000)
C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:336 +0x89
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0x0, 0x0}, {0xe248e3, ...}, ...})
C:/workdir/go/src/cmd/link/internal/ld/main.go:269 +0xef2
main.main()
C:/workdir/go/src/cmd/link/main.go:72 +0xedb
Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "golang.org/x/text/internal/triegen.test" && test == ""
2022-10-14 17:33 windows-amd64-race text@1bdb400f go@bfcc3a75 x/text/internal/triegen.test [build] (log)
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x21 pc=0xb91b7a]
goroutine 1 [running]:
cmd/link/internal/loader.(*Loader).OuterSym(...)
C:/workdir/go/src/cmd/link/internal/loader/loader.go:1695
cmd/link/internal/ld.(*deadcodePass).flood(0xc00006aa00)
C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:239 +0x265
cmd/link/internal/ld.deadcode(0xc0000c0000)
C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:336 +0x89
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0x0, 0x0}, {0xe248e3, ...}, ...})
C:/workdir/go/src/cmd/link/internal/ld/main.go:269 +0xef2
main.main()
C:/workdir/go/src/cmd/link/main.go:72 +0xedb
@bcmills
There are couple of different stack traces.
I looked at the first stack trace in the log:
go build golang.org/x/text/internal/number: # golang.org/x/text/internal/number [golang.org/x/text/internal/number.test]
unexpected fault address 0xffffffffffffffff
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x4fe86]
goroutine 1 [running]:
runtime.throw({0x9d37da?, 0xc00011c4a8?})
C:/workdir/go/src/runtime/panic.go:1047 +0x65 fp=0xc00011c470 sp=0xc00011c440 pc=0x79e85
runtime.sigpanic()
C:/workdir/go/src/runtime/signal_windows.go:270 +0xd0 fp=0xc00011c4b8 sp=0xc00011c470 pc=0x8ed50
runtime.mapassign(0x95cbe0, 0xc000113e00, 0x755a2d?)
C:/workdir/go/src/runtime/map.go:621 +0x246 fp=0xc00011c540 sp=0xc00011c4b8 pc=0x4fe86
cmd/compile/internal/ssagen.InitTables.func3({0x9dcb21, 0xb}, {0x9dbcfe, 0xb}, {0x9e9cf8, 0x17}, {0x9d2ad8, 0x5}, {0xc000126580, 0xe, ...})
C:/workdir/go/src/cmd/compile/internal/ssagen/ssa.go:3944 +0x1fa fp=0xc00011c608 sp=0xc00011c540 pc=0x74a25a
cmd/compile/internal/ssagen.InitTables()
C:/workdir/go/src/cmd/compile/internal/ssagen/ssa.go:4755 +0xb054 fp=0xc00011da98 sp=0xc00011c608 pc=0x72e754
cmd/compile/internal/gc.Main(0xa0cfd8)
C:/workdir/go/src/cmd/compile/internal/gc/main.go:173 +0xaaa fp=0xc00011df20 sp=0xc00011da98 pc=0x8e672a
main.main()
C:/workdir/go/src/cmd/compile/main.go:57 +0xdd fp=0xc00011df80 sp=0xc00011df20 pc=0x90e93d
runtime.main()
C:/workdir/go/src/runtime/proc.go:250 +0x1fe fp=0xc00011dfe0 sp=0xc00011df80 pc=0x7c5fe
runtime.goexit()
C:/workdir/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0xaca81
The crash happens here
https://github.com/golang/go/blob/bfcc3a755cbd9b5c38a4961f1df76c4db0d41442/src/runtime/map.go#L621
Which looks like uninitialised map used in
https://github.com/golang/go/blob/bfcc3a755cbd9b5c38a4961f1df76c4db0d41442/src/cmd/compile/internal/ssagen/ssa.go#L3944
I have no idea how to fix that problem.
Alex
This crash happens early on during compiler startup, before it has even read any of the code to be compiled. In fact, I think the compiler is completely deterministic up to this point, so it is hard to imagine why it would crash on these runs but not every time. So some unlikely but possible causes: something wrong with runtime/gc, randomness in map hash triggering some rare map bug, or our favorite boogeyman, bad machine memory.
Both failures are with -race, which may implicate the race detector itself or our instrumentation that calls it (e.g. if it messed up live variable stack maps that in turn caused the GC to overcollect).
The crashes are in the compiler and linker. Just double checking, do we build the toolchain with the race detector enabled on race builders, or just the tests?
Good point, I don't think the compiler/linker is race-enabled when on a race builder. Just the resulting tests.
In triage the consensus is to wait for more to happen. @prattmic notes that it's suspicious that 3 compiler/linker processes crashed at once. Maybe something wrong with the machine?
Found new dashboard test flakes for:
#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`
2022-09-29 20:41 windows-386-2008 sync@8fcdb60f go@b7662047 x/sync/syncmap.TestMapMatchesDeepCopy (log)
unexpected fault address 0xb4df0b0
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xb4df0b0 pc=0x40eaea]
runtime.throw({0x533c48, 0x5})
C:/workdir/go/src/runtime/panic.go:1047 +0x4d fp=0xa8439c0 sp=0xa8439ac pc=0x43947d
runtime.sigpanic()
C:/workdir/go/src/runtime/signal_windows.go:270 +0x105 fp=0xa8439e4 sp=0xa8439c0 pc=0x44d395
runtime.evacuated(...)
C:/workdir/go/src/runtime/map.go:204
...
C:/workdir/go/src/runtime/asm_386.s:541 +0x30 fp=0xa843c00 sp=0xa843bdc pc=0x467f40
reflect.Value.call({0x5198a0, 0x540e88, 0x13}, {0x5339d3, 0x4}, {0xa969390, 0x1, 0x1})
C:/workdir/go/src/reflect/value.go:586 +0x894 fp=0xa843e9c sp=0xa843c00 pc=0x4a79a4
reflect.Value.Call({0x5198a0, 0x540e88, 0x13}, {0xa969390, 0x1, 0x1})
C:/workdir/go/src/reflect/value.go:370 +0x7e fp=0xa843ed8 sp=0xa843e9c pc=0x4a707e
testing/quick.CheckEqual({0x5198a0, 0x540e8c}, {0x5198a0, 0x540e88}, 0x0)
C:/workdir/go/src/testing/quick/quick.go:331 +0x314 fp=0xa843f74 sp=0xa843ed8 pc=0x505994
golang.org/x/sync/syncmap_test.TestMapMatchesDeepCopy(0xa8312c0)
C:/workdir/gopath/src/golang.org/x/sync/syncmap/map_test.go:108 +0x47 fp=0xa843f9c sp=0xa843f74 pc=0x507b47
testing.tRunner(0xa8312c0, 0x540e80)
That latest failure looks unrelated, maybe?
Nothing -race about it.
There do seem to be 2 simultaneous failures in the log (like was observed above, I think), the linker crashing on a bad map operation and a golang.org/x/sync/syncmap test also crashing similarly.
This failure was also 386, not amd64.
No obvious culprit here, unfortunately.
The failure looks like machine issue and there doesn't seem actionable for us for the moment. Move the backlog and see if we have failures.
Timed out in state WaitingForInfo. Closing.
(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)
Found new dashboard test flakes for:
#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`
2023-05-03 20:35 linux-ppc64-sid-power10 tools@ba892bba go@c746059a x/tools/go/analysis/passes/ifaceassert.Test (log)
unexpected fault address 0x8000314bfe3e4a0c
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x3 addr=0x8000314bfe3e4a0c pc=0x181cc]
runtime.gopark(0x4d3c80?, 0xc0002e9b00?, 0x0?, 0x0?, 0x7fffae5e0108?)
/workdir/go/src/runtime/proc.go:398 +0x114 fp=0xc000093808 sp=0xc0000937d8 pc=0x4d5e4
runtime.goparkunlock(...)
/workdir/go/src/runtime/proc.go:404
runtime.semacquire1(0xc0002ea458, 0x0?, 0x1, 0x0, 0x0?)
/workdir/go/src/runtime/sema.go:160 +0x244 fp=0xc000093870 sp=0xc000093808 pc=0x60c04
sync.runtime_Semacquire(0xc000083520?)
/workdir/go/src/runtime/sema.go:62 +0x3c fp=0xc0000938b8 sp=0xc000093870 pc=0x7f43c
sync.(*WaitGroup).Wait(0xc0002ea450)
/workdir/go/src/sync/waitgroup.go:116 +0x88 fp=0xc0000938e8 sp=0xc0000938b8 pc=0x8ae18
golang.org/x/tools/go/packages.(*loader).refine(0xc0001181c0, 0xc0000b59f0)
/workdir/gopath/src/golang.org/x/tools/go/packages/packages.go:778 +0xd98 fp=0xc000093bf0 sp=0xc0000938e8 pc=0x26bc18
golang.org/x/tools/go/packages.Load(0xc0000aec60?, {0xc0000d6440, 0x2, 0x2})
/workdir/gopath/src/golang.org/x/tools/go/packages/packages.go:266 +0xbc fp=0xc000093c38 sp=0xc000093bf0 pc=0x269f5c
golang.org/x/tools/go/analysis/analysistest.loadPackages(0x4ca4a0, {0xc0000e8280, 0x4e}, {0xc0000d6440, 0x2, 0x2})
/workdir/gopath/src/golang.org/x/tools/go/analysis/analysistest/analysistest.go:332 +0x1f4 fp=0xc000093db0 sp=0xc000093c38 pc=0x276b34
golang.org/x/tools/go/analysis/analysistest.Run({0x355b88, 0xc000083380}, {0xc0000e8280, 0x4e}, 0x4ca4a0, {0xc0000d6440, 0x2, 0x2})
/workdir/gopath/src/golang.org/x/tools/go/analysis/analysistest/analysistest.go:286 +0xc8 fp=0xc000093e90 sp=0xc000093db0 pc=0x276608
golang.org/x/tools/go/analysis/passes/ifaceassert_test.Test(0x0?)
/workdir/gopath/src/golang.org/x/tools/go/analysis/passes/ifaceassert/ifaceassert_test.go:21 +0xe4 fp=0xc000093f30 sp=0xc000093e90 pc=0x279524
testing.tRunner(0xc000083380, 0x30a108)
Found new dashboard test flakes for:
#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`
2023-05-08 21:24 linux-ppc64-sid-power10 oauth2@839de225 go@4406e8ef runtime [build] (log)
unexpected fault address 0x17aef00000000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x17aef00000000 pc=0x17aef00000000]
goroutine 3031 [running]:
runtime.throw({0x96d141?, 0xc000c65e40?})
runtime/panic.go:1077 +0x40 fp=0xc0010b4920 sp=0xc0010b48e0 pc=0x58ff0
runtime.sigpanic()
runtime/signal_unix.go:866 +0x264 fp=0xc0010b4990 sp=0xc0010b4920 pc=0x73ae4
cmd/internal/src.XPos.IsKnown(...)
...
cmd/compile/internal/ssagen.Compile(0xc001a80b00, 0x81bc50?)
cmd/compile/internal/ssagen/pgen.go:187 +0x48 fp=0xc0010b5f30 sp=0xc0010b5d60 pc=0x664408
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
cmd/compile/internal/gc/compile.go:184 +0x4c fp=0xc0010b5f80 sp=0xc0010b5f30 pc=0x86a78c
cmd/compile/internal/gc.compileFunctions.func3.1()
cmd/compile/internal/gc/compile.go:166 +0x48 fp=0xc0010b5fc0 sp=0xc0010b5f80 pc=0x86ab78
runtime.goexit()
runtime/asm_ppc64x.s:912 +0x4 fp=0xc0010b5fc0 sp=0xc0010b5fc0 pc=0x956e4
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 177
cmd/compile/internal/gc/compile.go:165 +0x20c
2023-05-31 19:15 linux-ppc64le-buildlet go@3cea9e68 runtime [build] (log)
unexpected fault address 0x216e000004c18
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x3 addr=0x216e000004c18 pc=0x216e000004c18]
goroutine 2836 [running]:
runtime.throw({0x95d17f?, 0x68b614?})
runtime/panic.go:1077 +0x40 fp=0xc000a52000 sp=0xc000a51fc0 pc=0x59260
runtime.sigpanic()
runtime/signal_unix.go:866 +0x260 fp=0xc000a52070 sp=0xc000a52000 pc=0x73af0
cmd/compile/internal/ssagen.(*state).stmtList(...)
...
runtime.goexit()
runtime/asm_ppc64x.s:912 +0x4 fp=0xc000a53fc0 sp=0xc000a53fc0 pc=0x95034
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 143
cmd/compile/internal/gc/compile.go:165 +0x20c
goroutine 2822 [running]:
goroutine running on other thread; stack unavailable
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 143
cmd/compile/internal/gc/compile.go:165 +0x20c
go tool dist: FAILED: /workdir/go/pkg/tool/linux_ppc64le/go_bootstrap install cmd: exit status 1
Found new dashboard test flakes for:
#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`
2023-11-08 21:52 linux-amd64-longtest-race benchmarks@444331c9 go@3073f3f9 x/benchmarks/sweet/cmd/sweet.TestSweetEndToEnd (log)
--- FAIL: TestSweetEndToEnd (266.19s)
integration_test.go:36: phase setup @0s (duration: 13.80441237s)
integration_test.go:36: phase sweet-get @13.80441237s (duration: 39.512148632s)
integration_test.go:131: phase sweet-run-biogo-igor @53.317580879s (duration: 1m43.128894734s)
integration_test.go:131: phase sweet-run-biogo-krishna @53.317240974s (duration: 1m55.915230187s)
integration_test.go:169: output for go-build:
integration_test.go:177:
integration_test.go:183: command output:
[sweet] Work directory: /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-1
[sweet] Benchmarks: go-build (1 runs, 1 pgo runs)
...
<autogenerated>:1: internal compiler error: '.eq.internal/coverage/rtcov.CovMetaBlob': panic during prove while compiling .eq.internal/coverage/rtcov.CovMetaBlob:
runtime error: invalid memory address or nil pointer dereference
goroutine 21 [running]:
cmd/compile/internal/ssa.Compile.func1()
../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/ssa/compile.go:49 +0x6c
panic({0xd7ac00?, 0x14896f0?})
../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/runtime/panic.go:763 +0x132
cmd/compile/internal/ssa.(*factsTable).restore(...)
...
cmd/compile/internal/ssagen.Compile(0xc0031330e0, 0x1)
../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:216 +0x3e fp=0xc00256df70 sp=0xc00256dea0 pc=0xab19fe
cmd/compile/internal/gc.compileFunctions.func5.1(0xc0003f0280?)
../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:182 +0x34 fp=0xc00256dfb0 sp=0xc00256df70 pc=0xcf4054
cmd/compile/internal/gc.compileFunctions.func3.1()
../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:164 +0x30 fp=0xc00256dfe0 sp=0xc00256dfb0 pc=0xcf4470
runtime.goexit({})
../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc00256dfe8 sp=0xc00256dfe0 pc=0x47dd41
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 33
../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:163 +0x247