SpacetimeDB
SpacetimeDB copied to clipboard
clockworklabs
Better module backtraces for panics and whatnot
Description of Changes
Expected complexity level and risk
1 - it's pretty self-contained, and backwards-compatible with the existing logs data format
I'd prefer we used something like color-eyre for formatting which already handles panics instead of rolling something custom we'd have to maintain.
I just want to double check if the parsing and color formatting is happening in module code or not.
@RReverser does color-eyre panic stuff work in wasm? From what I can tell it doesn't expose the standalone backtrace formatting functionality, and even if it did it only supports the actual native backtrace::Backtrace type. (color-backtrace does expose it, but has the same thing with Backtrace). We're in a weird situation here cause wasm doesn't have a means to walk the call stack or even access the DWARF symbols itself, so we have to depend on wasmtime to do it for us. So, we can get a WasmBacktrace, which no crates like color-eyre can actually consume themselves, and we also have to serialize it to store it in the logs anyway. I chose to do the filtering and formatting on the client side rather than the server side so that there's less overhead in the console_log syscall, and so that possibly important information (e.g. for a weird bug somehow happening in omitted frames) doesn't get lost permanently and can just be configured to be shown in the client.
@coolreader18 Does Wasmtime attaching symbols not help with backtrace? If it helps with gdb/lldb, there should be a way to make Rust backtraces use the same symbol resolution so that all these backtrace formatters/tools "just work".
Another reason I'd really prefer not to do this on module side is that it further diverges support between module languages. With AOT we'll be able to get symbol names for C# functions in Wasm too, which will work correctly work with wasm2wat or debuggers, but still won't provide readable backtraces with this approach because it's doing something Rust-specific only in Rust modules.
Ideally Wasm modules should be as opaque to the host as possible, and treated the same regardless of the language they were compiled in, especially if we add more langs down the line, otherwise we'll end up manually replicating same functionality for each language individually.
No, wasm has no mechanism to access the call stack or the DWARF section. backtrace-rs on wasm32 unconditionally selects a noop implementation. And this isn't rust-specific at all? This would work with any language that generates debug symbols, we'd just need to possibly add a tiny bit of support to the cli to make it display nicer.
No, wasm has no mechanism to access the call stack or the DWARF section.
Wasm doesn't, but the native stack walker in backtrace should be able to resolve JIT functions just like it resolves any native symbols via debug info, now that we have Wasmtime which attaches said debug info. I can try to look into this myself later.
And this isn't rust-specific at all?
The changes in bindings crate - e.g. __rust_{begin,end}_short_backtrace - are by definition Rust-specific, as only Rust modules use it. And then in log formatter you're looking for those __rust_{begin,end}_short_backtrace which suggests it would behave differently for Wasm that doesn't have those functions.
Bringin this one back, I guess. @RReverser you're correct that these changes are rust-specific, but the bindings crate changes are rust-specific because the bindings crate is rust-specific. If csharp has similar frames to __rust_{begin,end}_short_backtrace, it's a two line change to add those in cli.
