[question] Ever-increasing rate of allocations

[tsmethurst]

Hello hello! Firstly I wanna say a big thank you for wazero, it's enabled a lot of really funky stuff for GoToSocial such as running ffmpeg as a wasm module, which has been really exciting to implement!

Unfortunately despite all that wazero brings to GtS, we're having a bit of trouble with one particular issue as we're ironing out the implementation on the way to our first release including this code. Specifically, we've noticed recently in our metrics on a running GoToSocial instance that the rate of allocations seems to increase slowly over time. For example, in the following graph:

Here you can see that over a period of a couple of days, the rate at which objects are allocated seems to increase. This leads to spikier memory usage overall, increased garbage collection, and eventually OOM errors as the OS kills the GtS process.

For comparison, here's a screenshot of the allocations on a previous version of GoToSocial before we started using wazero for ffmpeg.

To try to figure out what's happening, I've been looking through our allocs via pprof and the results have been frustratingly tight-lipped in terms of what the issue might actually be. I also tried running some tests where I repeatedly reprocess images, but I can't seem to reproduce the problem on a testbed.

So I'm wondering, do you all have any idea what might be happening here, and know of any reason why the rate of allocations might be increasing over time in this way? For a bit of context, this is how we currently instantiate wazero modules: https://github.com/superseriousbusiness/gotosocial/blob/main/internal/media/ffmpeg/wasm.go. Ie., create the runtime once, compile the module once, and then instantiate a module from the compiled module every time we run.

Thanks for reading :)