gotify
[RFC] A plugin system that does not rely on Go plugin/dynamic linker
Is your feature request related to a problem? Please describe.
The current plugin system is built 5+ years ago with the Go dynamic loader when options were more limited. Building Go plugins were complicated, has limited OS support (requires glibc or MacOS) and runtime loading errors can be very difficult to debug.
Describe the solution you'd like
I still think a binary plugin system can be useful as backward compatibility/escape hatch, but my observations are most plugins are designed for basic message piping (like my broadcast plugin) or message format support.
I propose we transition the current plugin architecture to be IPC based (we execute the plugin file and communicate with it using sockets). This gives us:
- Backwards compatibility: existing plugins can just add a generic
mainfunction we provide , build as an executable and work. - Security isolation: if there are security vulnerabilities in the plugin the gotify process itself has more isolation. Moreover we can potentially enable the use of OS features (seccomp-bpf, etc) for users to load plugins with limited permissions.
Most of the APIs are easy to change with the exception of Webhooker, I am thinking about run a different instance of Gin on the plugin side and just use a mux to make the plugin answer for that HTTP request over the socket. This will wipe the remote address information from the object, we can try to emulate it back in our provided compatibility layer or make it a breaking change.
Describe alternatives you've considered
I have some other ideas but it seems be too drastic/too specialized for a specific situation. Feel free to disagree or improve on that!
- A gojq based message transformation program per app.
- Transition the whole Plugin architecture to WASM (maybe later, but I think keeping native code execution for compatibility and get us out of this dependency matching situation is more useful). This allows for plugins to be written in any language, loaded and unloaded on demand and super easy runtime permission control but it is not native code so will lose backwards compatibility
I actually don't even think existing plugin code needs to be changed at all, if it is a shared library we can just embed the go-plugin-to-IPC shim into the current plugin/compat package, fork ourselves and load it. I expect pretty minimal disruptions to existing workflows.
cc @jmattheis
I actually don't even think existing plugin code needs to be changed at all, if it is a shared library we can just embed the go-plugin-to-IPC shim into the current plugin/compat package, fork ourselves and load it. I expect pretty minimal disruptions to existing workflows.
Do we then not have the same problem with dependencies as it relies on the go plugin mechanism?
Would you see this as final version of the api system, or do you plan to move to WASM. I think if this is a temporary solution I'd rather do it correctly and don't "waste" time implementing the temporary solution.
@jmattheis
Do we then not have the same problem with dependencies as it relies on the go plugin mechanism?
Yes, it's a separate program so we communicate via gRPC/HTTP, etc over sockets (and maybe over network as well, it's more of a should we not a can we problem). The compatibility problem is more about API compatibility (which protobuf already has a solution with very good backwards compatibility).
I'd rather do it correctly and don't "waste" time implementing the temporary solution.
I am not really specifically looking forward to switch to WASM just because (hence I listed it as alternative solutions because it is a new route some projects use because of the reasons I listed).
I know you favor stability but the reality for WASI is still ABIs are changing quickly (there is WASIP1, WASIP2, WASIP1+threads, etc and they are not binary compatible), so I think it is not going to be the "final" version even if we go for that route right now.
If WASM is a problem for some users I can write a plugin shim that loads WASM files and delegate it as a normal plugin. We don't have to deal with that here.
Hashicorp uses the IPC/RPC route with pretty long history to reference: https://developer.hashicorp.com/vault/docs/plugins/plugin-development
I mean this statement for my first question.
if it is a shared library we can just embed the go-plugin-to-IPC shim into the current plugin/compat package
If we include this in the compat package which is inside gotify/server, then we do have to load the go plugin via plugin open, or how would this shim work.
I understand if the plugin would be a executable then we can just execute it by gotify/server and we don't have to do plugin.open so no dependency problem.
If WASM is a problem for some users I can write a plugin shim that loads WASM files and delegate it as a normal plugin. We don't have to deal with that here.
Good point, so basically IPC would be the main plugin system, and we could do a plugin that loads wasm to get wasm support without having any wasm code inside gotify/server.
Given this I'm okay with the IPC solution. Seems future proof, without the problems we have with the current go plugin solution.
If we include this in the compat package which is inside gotify/server, then we do have to load the go plugin via plugin open, or how would this shim work.
This is for backwards compatibility so the idea is:
- Older plugins (shared library form) will still work unless they rely on being in the same process as gotify/server by us forking ourselves and act like a shim. Another option is to have two different ways of loading plugins, I don't like it for consistency reasons.
- Newer plugins are just built as executable and we pass a file descriptor and communicate directly over IPC, there are no shims.
To me it is very important that gotify would be the process that creates and keeps track of the socket and starts and stops the plugin processes. I propose the use of unix sockets and to pass the location to the plugin by command line arguments at the start.
I am currently trying to gear towards gRPC and keeping our options open on exactly which transport to use, but unix socket is on my top choice list.
My current architecture is we will no longer have callback registrations and simply give the plugin a token to send message or store data through the server listener.
And the other server-to-plugin calls (displayer, configurer, etc) will go through a shared RPC socket with gRPC.
Webhooks will be reverse proxied via a different socket just in case (security mishaps, etc), basically the idea is if the path satisfies startsWith(pluginToken) we send it through the socket.
I posted my current WIP here: https://github.com/gotify/plugin-api/pull/10
I just wanted to add my opinions after building a plugin with the existing system.
For the WASM direction, I would be worried about how much this complicates the plugin build system. I feel like you would run into similar issues to my next point.
The Go plugin system is a little awkward to build around, I ran into some overlapping dependency issues and troubles getting the builds to match the version of the Gotify server.
The gRPC direction sounds quite good, it would be nice to have a the bi-directional communication. That would be a much nicer interface than connecting to the websocket.
Having the plugin decoupled from the server application would be nice for development. The development cycle of compiling the plugin, copying it to the server and restarting it is a little tedious. It would be nice to just point it at a socket and restart it at will.
Being decoupled should also make it easier to build integration tests around plugins.
I think having the Gotify server manage the executables of the plugins make sense. It would also be nice if I could build a plugin as an external service. Then I could build a deeper integration into a larger application.
Yeah I agree WASM is more of an acknowledgement for a novel approach that some projects are using. I don't think it is strictly needed (actually you probably can hack something together with binfmt_misc that make it work even without us supporting it.
I am pretty committed to the gRPC direction at this point with HashiCorp's model (Plugin as gRPC server) as inspiration. Regarding executable management, I am not yet fully committed on registration (should we leave remote plugins over the wire on the table? I am not currently sure and I am gearing towards a no at this point).
I am thinking about fixing #203 as well in this new plugin API edition where I provide a way for a plugin to register a client session on the server's behalf.