wasmtime-go
wasmtime-go copied to clipboard
sigill on linux/arm64 (emulated)
With this main.go (files here):
package main
import (
"log"
"time"
"github.com/bytecodealliance/wasmtime-go"
)
func main() {
bs, err := wasmtime.Wat2Wasm(`
(func (export "loop")
(loop br 0))
`)
check(err)
for i := 0; ; i++ {
config := wasmtime.NewConfig()
config.SetInterruptable(true)
store := wasmtime.NewStore(wasmtime.NewEngineWithConfig(config))
module, err := wasmtime.NewModule(store.Engine, bs)
check(err)
instance, err := wasmtime.NewInstance(store, module, nil)
check(err)
handle, err := store.InterruptHandle()
check(err)
go func() {
time.Sleep(100 * time.Millisecond)
handle.Interrupt()
}()
_, err = instance.GetFunc(store, "loop").Call(store)
if err == nil {
panic("expected error")
}
if t, ok := err.(*wasmtime.Trap); ok {
log.Printf("%d: trap: %s", i, t.Message())
}
}
}
func check(e error) {
if e != nil {
panic(e)
}
}
The following command instantly fails for me:
$ docker run -it -v $(pwd):/src -w /src --platform linux/arm64 golang:1 go run main.go
go: downloading github.com/bytecodealliance/wasmtime-go v0.33.1
SIGILL: illegal instruction
PC=0x555a80e0b0 m=0 sigcode=2
instruction bytes: 0x50 0x76 0x93 0x0 0x55 0x0 0x0 0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x0 0x0
goroutine 0 [idle]:
runtime: unknown pc 0x555a80e0b0
stack: frame={sp:0x555a80ce50, fp:0x0} stack=[0x5500001bc8,0x5500800be0)
runtime: unknown pc 0x555a80e0b0
stack: frame={sp:0x555a80ce50, fp:0x0} stack=[0x5500001bc8,0x5500800be0)
goroutine 1 [syscall]:
runtime.cgocall(0x64df70, 0x4000123bf8)
/usr/local/go/src/runtime/cgocall.go:156 +0x50 fp=0x4000123bb0 sp=0x4000123b70 pc=0x5b4bf0
github.com/bytecodealliance/wasmtime-go._Cfunc_wasmtime_func_call(0x12fb1d0, 0x400013a0c0, 0x0, 0x0, 0x0, 0x0, 0x400013a100)
_cgo_gotypes.go:2076 +0x44 fp=0x4000123bf0 sp=0x4000123bb0 pc=0x645d94
github.com/bytecodealliance/wasmtime-go.(*Func).Call.func1.1({0xf3d4c0, 0x4000118220}, 0x400013a0c0, 0x0, {0x12ebcc8, 0x0, 0x0}, 0x0, {0x12ebcc8, 0x0, ...}, ...)
/go/pkg/mod/github.com/bytecodealliance/[email protected]/func.go:430 +0xbc fp=0x4000123c50 sp=0x4000123bf0 pc=0x64886c
github.com/bytecodealliance/wasmtime-go.(*Func).Call.func1(0x400013a100)
/go/pkg/mod/github.com/bytecodealliance/[email protected]/func.go:430 +0xa8 fp=0x4000123cd0 sp=0x4000123c50 pc=0x648788
github.com/bytecodealliance/wasmtime-go.enterWasm({0xf3d4c0, 0x4000118220}, 0x4000123e48)
/go/pkg/mod/github.com/bytecodealliance/[email protected]/func.go:513 +0x60 fp=0x4000123d20 sp=0x4000123cd0 pc=0x648a70
github.com/bytecodealliance/wasmtime-go.(*Func).Call(0x400013a0c0, {0xf3d4c0, 0x4000118220}, {0x0, 0x0, 0x0})
/go/pkg/mod/github.com/bytecodealliance/[email protected]/func.go:414 +0x1b8 fp=0x4000123ea0 sp=0x4000123d20 pc=0x648078
main.main()
/src/main.go:35 +0x1c4 fp=0x4000123f70 sp=0x4000123ea0 pc=0x64d114
runtime.main()
/usr/local/go/src/runtime/proc.go:255 +0x284 fp=0x4000123fd0 sp=0x4000123f70 pc=0x5e7a64
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1133 +0x4 fp=0x4000123fd0 sp=0x4000123fd0 pc=0x613564
r0 0x4000000000
r1 0x0
r2 0x4000002000
r3 0x55009379d8
r4 0x0
r5 0x17f
r6 0x2000555a7ee023
r7 0x6474e550
r8 0x84
r9 0x4
r10 0x1
r11 0x0
r12 0x12fdbe0
r13 0x55009379d8
r14 0x0
r15 0xffffffff
r16 0x1
r17 0x5500908a40
r18 0x0
r19 0x0
r20 0x555a80e580
r21 0x555a80e1c0
r22 0x0
r23 0xb24924
r24 0xffffffffffffffff
r25 0x5500800990
r26 0x1
r27 0x1
r28 0x10
r29 0x555a80e0a0
lr 0x555a80e0b0
sp 0x555a80ce50
pc 0x555a80e0b0
fault 0xffffffffbf0ed000
exit status 2
It works fine and loops for a while if run using --platform linux/amd64
or if not using docker at all, on macos/amd64 host. Since qemu and binfmt_misc is involved when running a different platform, I don't know if the problem happens on a actual linux/arm64 hardware, or if this is somehow emulation-related.
Running the prograrm natively with go version go1.16.6 linux/arm64
it works alright for me, and in docker on the same machine it unfortunately doesn't get past this line. My guess though is that this is likely related to one of the emulation layers involved?
@alexcrichton What is your kernel version? For example, Ubuntu 18.04 (which, unfortunately in this case, is still supported) by default ships with a kernel that does not support the variant of the membarrier()
system call you are pointing at (i.e. the kernel is older than 4.16).
My uname -a
reports as:
Linux arm2-ci.infra.bytecodealliance.org 4.19.0-9-arm64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) aarch64 GNU/Linux
So I think it may be supported? I'll note that the membarrier
call only fails inside of docker for me, natively it works just fine (not sure what's happening there, I'm just sort of naively assuming it's some docker thing)
You are probably right - Docker stopped blocking membarrier()
by default less than 2 years ago. You can try this suggestion from the documentation to unblock it.
Ah perfect! Pasing --security-opt seccomp=unconfined
to docker run
I'm able to successfully run this program on Linux arm64 hardware, even in docker. That may mean @srenatus that the issue here is Docker's arm64 emulation on x86_64? Or I'm not actually sure, what is the host system you're running on when you're passing --platform linux/arm64
to docker and seeing the crash?
I'm not actually sure, what is the host system you're running on when you're passing --platform linux/arm64 to docker and seeing the crash?
Host is macos 12.1 (Darwin Kernel Version 21.2.0 Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64
), running docker desktop 4.1.1... docker tells me this much about its kernel:
$ docker run -it --platform linux/amd64 alpine uname -a
Linux fe90a75cad3e 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 Linux
$ docker run -it --platform linux/arm64 alpine uname -a
Linux dbab877dfae4 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 aarch64 Linux
Ah ok, looks like you're emulating linux-arm64 on macos-x86_64. AFAIK that's a pretty serious emulation layer and given that the issue only happens there I'd probably chalk it up to the emulation layer for now.
FWIW I've noticed this originally emulating arm64 on Linux/amd64, in a GitHub action run. But I've got little control over that environment, so I've taken the laptop for reproducing the error. Probably still got something to do with the emulation involved... 🤷
@alexcrichton thanks for taking the time to look into this with me ❤️