Viceroy icon indicating copy to clipboard operation
Viceroy copied to clipboard
verified publisher icon fastly

Empty TinyGo project crashes under component adapter

Open erikrose opened this issue 6 months ago • 6 comments

A Fastly Compute project using the TinyGo SDK and having an empty main() crashes when run under viceroy --adapt:

% RUST_BACKTRACE=1 target/debug/viceroy --adapt /Users/me/Desktop/main\ -scheduler=none.wasm
unreachable executed at adapter line 1555: assertion failed

2025-06-05T17:56:20.354549Z ERROR WebAssembly trapped: error while executing at wasm backtrace:
    0: 0x173da - wit-component:adapter:wasi_snapshot_preview1!random_get
    1: 0x18797 - wit-component:shim!adapt-wasi_snapshot_preview1-random_get
    2:  0x459 - __wasi_random_get
                    at /Users/runner/work/tinygo/tinygo/lib/wasi-libc/libc-bottom-half/sources/__wasilibc_real.c:598:19
    3:  0x47a - __getentropy
                    at /Users/runner/work/tinygo/tinygo/lib/wasi-libc/libc-bottom-half/sources/getentropy.c:11:13
    4:  0xa01 - arc4random_buf
                    at /Users/runner/work/tinygo/tinygo/lib/wasi-libc/libc-top-half/sources/arc4random.c:101:13
    5:  0xfa3 - arc4random
                    at /Users/runner/work/tinygo/tinygo/lib/wasi-libc/libc-top-half/sources/arc4random.c:138:5
    6: 0x184d - runtime.hardwareRand
                    at /opt/homebrew/Cellar/tinygo/0.35.0/src/runtime/runtime_tinygowasm.go:106:29
              - runtime.init#1
                    at /opt/homebrew/Cellar/tinygo/0.35.0/src/runtime/algorithm.go:27:22
              - runtime.run
                    at /opt/homebrew/Cellar/tinygo/0.35.0/src/runtime/scheduler_none.go:16:9
              - _start
                    at /opt/homebrew/Cellar/tinygo/0.35.0/src/runtime/runtime_wasmentry.go:20:5
    7: 0x17bb6 - wit-component:adapter:wasi_snapshot_preview1!fastly:api/reactor#serve

Caused by:
    wasm trap: wasm `unreachable` instruction executed

Stack backtrace:
   0: std::backtrace_rs::backtrace::libunwind::trace
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2: std::backtrace::Backtrace::create
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/backtrace.rs:331:13
   3: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.98/src/backtrace.rs:27:14
   4: <T as core::convert::Into<U>>::into
             at /Users/me/.rustup/toolchains/1.83-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/convert/mod.rs:759:9
   5: <T as wasmtime_types::prelude::IntoAnyhow>::into_anyhow
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-types-25.0.3/src/prelude.rs:74:9
   6: wasmtime::runtime::trap::from_runtime_box
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-25.0.3/src/runtime/trap.rs:118:34
   7: wasmtime::runtime::func::invoke_wasm_and_catch_traps::{{closure}}
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-25.0.3/src/runtime/func.rs:1610:28
   8: core::result::Result<T,E>::map_err
             at /Users/me/.rustup/toolchains/1.83-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:856:27
   9: wasmtime::runtime::func::invoke_wasm_and_catch_traps
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-25.0.3/src/runtime/func.rs:1610:9
  10: wasmtime::runtime::func::Func::call_unchecked_raw
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-25.0.3/src/runtime/func.rs:1072:9
  11: wasmtime::runtime::component::func::Func::call_raw
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-25.0.3/src/runtime/component/func.rs:468:13
  12: wasmtime::runtime::component::func::typed::TypedFunc<Params,Return>::call_impl
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-25.0.3/src/runtime/component/func/typed.rs:207:17
  13: wasmtime::runtime::component::func::typed::TypedFunc<Params,Return>::call_async::{{closure}}::{{closure}}
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-25.0.3/src/runtime/component/func/typed.rs:189:31
  14: wasmtime::runtime::store::<impl wasmtime::runtime::store::context::StoreContextMut<T>>::on_fiber::{{closure}}::{{closure}}
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-25.0.3/src/runtime/store.rs:2107:34
  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /Users/me/.rustup/toolchains/1.83-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2454:9
  16: wasmtime_fiber::Suspend<Resume,Yield,Return>::execute::{{closure}}
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-fiber-25.0.3/src/lib.rs:201:62
  17: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /Users/me/.rustup/toolchains/1.83-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9
  18: std::panicking::try::do_call
             at /Users/me/.rustup/toolchains/1.83-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:557:40
  19: std::panicking::try
             at /Users/me/.rustup/toolchains/1.83-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:520:19
  20: std::panic::catch_unwind
             at /Users/me/.rustup/toolchains/1.83-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panic.rs:358:14
  21: wasmtime_fiber::Suspend<Resume,Yield,Return>::execute
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-fiber-25.0.3/src/lib.rs:201:22
  22: wasmtime_fiber::unix::fiber_start
             at /Users/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasmtime-fiber-25.0.3/src/unix.rs:207:9
  23: _wasmtime_fiber_start_25_0_3
2025-06-05T17:56:20.376644Z ERROR There was an error handling the request error while executing at wasm backtrace:
    0: 0x173da - wit-component:adapter:wasi_snapshot_preview1!random_get
    ...same backtrace as up top.

Note that you need to pull in https://github.com/fastly/Viceroy/pull/489 to get code to successfully run under the adapter at all.

Truths

  • The assertion refers to the corruption of the magic2 magic number in the State struct.
  • It can't be blamed on the scheduler, as it crashes even when the build is done with -scheduler=none. It is triggered by the presence of GC: a build using -scheduler=none -gc=leaking does not crash. (These can both be adjusted in fastly.toml's [scripts] section.)
  • Building the same project with Big Go does not crash. (GOARCH=wasm GOOS=wasip1 go build -o bin/main.wasm ./)

erikrose avatar Jun 09 '25 15:06 erikrose

There's a decent chance that this is caused by this line in tinygo.

The component adapter is a Wasm module that doesn't define its own memory; it imports the memory of the main module. To claim some of the memory for its own use, the adapting process increases the size of the main module's linear memory, making it look as if something had done a memory.grow. In theory, language runtimes should allow user code to do memory.grow at any time, but it's a little tricky in the adapter's situation because there, the grow happens before the program starts executing. That means calling memory.size at the very beginning of the program and assuming that all memory up to that size is unused doesn't work.

One way to fix this would be to have tinygo do something like the __heap_end symbol trick that wasi-libc uses.

dgohman-fastly avatar Jun 09 '25 15:06 dgohman-fastly

Ah, the issue appears to be more involved. If I understand this code correctly, TinyGo doesn't support users calling memory.grow, as it assumes its heap is contiguous.

wasm-tools' adapter process supports calling cabi_realloc in the main module instead of doing a raw memory.grow, if cabi_realloc exists, however it doesn't appear that tinygo current exports a cabi_realloc. So wasm-tools is probably falling back to its polyfill which uses memory.grow.

dgohman-fastly avatar Jun 09 '25 16:06 dgohman-fastly

wasm-reduce yields this shorter version which still crashes. I've hand-stripped the producers and target_features sections off.

(module
  (type (;0;) (func))
  (type (;1;) (func (param i32 i32) (result i32)))
  (import "wasi_snapshot_preview1" "random_get" (func (;0;) (type 1)))
  (memory (;0;) 2)
  (export "memory" (memory 0))
  (export "_start" (func 2))
  (func (;1;) (type 0)
    i32.const 0
    i32.const 0
    call 0
    drop
  )
  (func (;2;) (type 0)
    (local i32 i32)
    i32.const 66364
    memory.size
    i32.const 16
    i32.shl
    i32.store
    i32.const 66496
    i32.const 66364
    i32.load
    i32.const 1
    i32.sub
    i32.store
    i32.const 66496
    i32.load
    i32.const 0
    i32.const 1
    memory.fill
    call 1
  )
)

Reduction script

erikrose avatar Jun 09 '25 17:06 erikrose

wasm-reduce is a bit aggressive and randomly mutates instructions. In our case, where the State lives in wasm linear memory, it can quite possibly land on an otherwise unrelated program that simply stores a value overtop of it. "Reduce" is a misnomer.

Here's a more proper but larger reduction from wasm-tools shrink.

erikrose avatar Jun 09 '25 21:06 erikrose

The adapter will happily use cabi_realloc(), if it can find it, at adaption time to carve out a chunk of linear memory for itself. As a test, Dan and I added…

//export cabi_realloc
func cabi_realloc(ptr, oldsize, align, newsize unsafe.Pointer) unsafe.Pointer {
	var space = make([]byte, int(100_000))
	return unsafe.Pointer(&space[0])
}

…to the crashing empty project, and the crash went away. Now we just need to put that over in the SDK, actually use newsize, and keep Go from GCing that RAM.

erikrose avatar Jun 09 '25 21:06 erikrose

Solutions I considered:

  1. Have the adapter add a cabi_realloc() which calls malloc(), which allocates a region of memory and hangs onto it from the GC's point of view. Pro: nice and clean, with no trouble calling across language boundaries due to the explicit contract of the malloc function. Con: TinyGo is trying to remove the export of malloc. However, the issue has been open for years, with no comments in the past year.
  2. Do the same as above, but call runtime.alloc, the lower-level routine which allocates memory but doesn't hang onto it. In order to not have it paved over by future allocations, we'd have to retain a reference to it in a stack frame that outlives the GC. We could smuggle a pointer to it into that frame by stowing it in a wasm global that we splice into the existing module. Pro: runtime.alloc might be more likely to exist in future TinyGo programs. Con: smooshing 2 independently compiled Go programs together (perhaps with different GCs) might not work reliably.

Pursuing number 1. Reasoning:

  1. Every TinyGo program today includes malloc. Thus, we should be able to adapt all our existing services reliably.
  2. If malloc is removed in a future release of TinyGo, we can update our contract to require that authors opt into its inclusion before pushing a new release. Failure to do so will be eagerly caught at service validation time.
  3. Ultimately, we will have our SDKs churn out only components, and no adaptation will be needed.

We'll invoke this functionality only if the module appears to have been generated by TinyGo (according to the producers section, most likely) and doesn't already provide a cabi_realloc().

erikrose avatar Jun 16 '25 21:06 erikrose