Performance degradation and crashes due to malloc/gc

[martijneken]

Hi folks, I'm looking for advice. We are building a product around ProxyWasm and we'd love to support as many languages/SDKs as possible. We've been doing benchmarking and stress testing on various plugins, including those built with TinyGo / ProxyWasmGoSdk. In benchmarking TinyGo we are seeing high malloc performance costs and even crashes.

My core questions are:

• what we can do to improve/tune malloc/gc in TinyGo?
• how can we debug GC behavior in TinyGo Wasm?
• how can we diagnose whether there are memory leaks / fragmentation?

This TinyGo documentation is worrisome:

Describe the bug / error

We see significant costs in malloc(), presumably due to GC. Here are CPU profiles of 2 plugin versions:

1. A plugin that compiles and executes Regexp in HTTP handlers. The full cost profile here is unclear but we expect there's a lot of GC happening because -print-allocs=. shows loads of heap allocations in the Regexp libs. If we run this plugin often enough (100 seconds, ~40k iterations), it crashes on malloc (stack below)... Crash stack:

ERROR from integration: Function: malloc failed: Uncaught RuntimeError: unreachable
Proxy-Wasm plugin in-VM backtrace:
  0:  0x6f19 - runtime.runtimePanicAt
  1:  0xbc9 - runtime.alloc
  2:  0x7eb6 - runtime.hashmapSet
  3:  0x745c - runtime.hashmapBinarySet
  4:  0x741f - malloc
  5:  0x14d7a - github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm.GetHttpRequestHeader
  6:  0x138d1 - proxy_on_request_headers

2. The same plugin, but optimized so Regexp compilation is moved to plugin init (outside the profile). The CPU cost here is dominated by malloc during a GetHttpRequestHeader call. This should not be expensive (compared to executing plugin logic including Regexps) so we suspect a memory management issue (presumably GC). The longer we run this plugin, the higher the per-invocation costs... Fragmentation? Memory leak?

What is your Envoy/Istio version?

N/A, benchmarks run with a shim ProxyWasm host implementation and v8 wasm engine.

What is the SDK version?

Go 1.22.5

What is your TinyGo version?

TinyGo 0.32.0

URL or snippet of your code including Envoy configuration

Customer proprietary, can't share exact code. I can create a min repro plugin if needed.