dim
dim copied to clipboard
Dusk-Labs
Design a plugin system.
Rhai looks really nice. This could definitely be used for plugins, however I think we should still look into using wasm instead.
WASM would allow plug-ins to use various languages, with near native performance. However, there are some limitations on WASM APIs. As it stands WASM bindings are limited to primitive numeric types and arrays of them. That might be OK for streaming audio or video, but it could be very awkward for complex structs. Also, calling to or from wasm requires all data to be copied to/from the wasm sandbox memory. Wasm would be great. These are just some things to consider.
There are some improvements to wasm-bindgen that allow passing arbitrary types between the host and a wasm guest. Serde is used to serialize the data during transfer between them. It's described here: Arbitrary Data with Serde.
There are some roadblocks to using the serde feature of wasm-bindgen. It creates a cyclic dependency on aHash whenever another crate (such as sqlx, Hashbrown or others) also depends on aHash. . That discussion has a recommendation, "I would recommend to basically avoid the serde features in wasm-bindgen.".
There is a lot of interest in resolving the issue. For the moment it may significantly limit the dependencies in the host application. Sounds like it might be resolved eventually.
I've thought quite a bit about how to write a extensible plugin system, and I think the best approach here is a RPC based plugin system where each plugin lives as its own process and manages its own resources and environment. The plugins could communicate with Dim using something like tarpc or capnproto. The maintenance burden would likely be the same as managing a C-ffi or wasm shim. We could also easily sandbox these plugins.
In regards to what kind of features I'd like to support, I'd like to add the ability for plugins to hook REST API calls as in the future I'd like to have a plugin that essentially would extend or replace the current streaming APIs with smarter versions with more features.
I'd also like to add the ability for plugins to register UI elements, so we could allow plugins to have their own dashboards or settings if necessary. This would most likely be only available on the Web-UI.
Binary RPCs sounds very efficient, leaving the door open for low latency interaction. Sounds great!
This is what is done for the Terraform plugin infrastructure. I would suggest taking a look at their writeup and implementation for inspiration.
Additionally, gRPC seems like a great approach for this. Here is a great example for reference.
