connect
connect copied to clipboard
redpanda-data
Play with a WASM processor
We should do a bit of explorative work and see how easily a WASM processor using https://github.com/suborbital/reactr would be.
tinygo was recently added, so it makes it much easier to test 👍 https://github.com/suborbital/reactr/tree/main/rwasm/testdata/tinygo-req
I know that there are exec type extensibility for inputs, processors, and outputs but the contract for these things are more expressive than a fifo of bytes, as the stdin/stdout interface of subprocess is.
This would make the flexibility of benthos endless, and let orgs more comfortable in other languages be able to use benthos as a platform for their code with lower entry cost. The full interface exposed in the public/service package can be exposed to the WASM implementation, and the WASM blob can be accessed from object store or whatever, which is a big advantage over building a go program to extend benthos.
Nothing new in this comment, just agreeing mostly. :)
I'm also very interested in WASM. I come from a Rust and JS background, and being able to write a WASM plugin would just be ideal.
The bytecode alliance have just released v1 of wasmtime, which will be a massive milestone for WASM and WASI. In a nutshell, startup time for a WASM invocation is now down to 5 microseconds.
There is a wasmtime-go package for easily embedding in a Go a program
Hey @cortopy, I forgot to update this ticket but there's an experimental WASI plugin implemented on this branch: https://github.com/benthosdev/benthos/tree/wasmtime-go, performance isn't amazing as we're having to pipe data to/from files but it's a starting point.
Ok it’s wasmtine. Was hoping it’s Wazero so there is zero cgo overhead / complications.
https://github.com/tetratelabs/wazero
@gedw99 wazero is literally the next thing I'm going to try, will aim to have both to choose from.
@gedw99 wazero is literally the next thing I'm going to try, will aim to have both to choose from.
Also checkout Capsule that uses Wazero and NATS together . I was thinking of adding Benthos into that stack for all data transforms.
All Actions are Sent to NATS workers ( wasm ) and benthos does any data transform required via a NATS stream topic.
The mutated data goes back into NATS out of Benthos kicking off another Benthos transform. Infinite recursion….
The key space in NATS is your address system.
You end up with a Google sheets self hosted. You get parametric recursion just like Google Sheets.
Okay so I've put together a wasm processor using wazero so we still avoid the cgo dependencies: https://github.com/benthosdev/benthos/commit/da7179aefb1d7dd32dc34a7e7243adab73f8be28, I decided in the end to implement helper functions for accessing/mutating message data. We'll need to expand the exported functions and grow the number of language examples but the basics are there and ready to try.
Closing this issue for now as the experimentation is concluded.
@Jeffail no CGO !!! that code looks really good.
if there is any examples ? would love to have a play..
@gedw99 Have a look here: https://github.com/benthosdev/benthos/tree/main/public/wasm I'm thinking to give Python a shot, but it looks like that might require some more work.
thansk @mihaitodor will look now..
am glad i dont have to do python to wasm.. But would be interesting to see if its works.