Add Steel as an optional plugin system

[mattwparas]

trafficstars

Notes:

• I still need to rebase up with the latest master changes, however doing so causes some headache with the lock file, so I'll do it after some initial feedback. Also, this depends on the event system in #8021.
• The large diff size is a combination of lock file changes + the dependency on the event system PR. The diff has ended up quite large with all of the other stuff
• I'm currently pointing to the master branch of steel as a dependency. This will point to a stable release on crates once I cut a release.

---

Opening this just to track progress on the effort and gather some feedback. There is still work to be done but I would like to gather some opinions on the direction before I continue more.

You can see my currently functioning helix config here and there are instructions listed in the STEEL.md file. The main repo for steel lives here, however much documentation is in works and will be added soon.

The bulk of the implementation lies in the engine.rs and scheme.rs files.

Design

Given prior conversation about developing a custom language implementation, I attempted to make the integration with Steel as agnostic of the engine as possible to keep that door open.

The interface I ended up with (which is subject to change and would love feedback on) is the following:

pub trait PluginSystem {
    /// If any initialization needs to happen prior to the initialization script being run,
    /// this is done here. This is run before the context is available.
    fn initialize(&self) {}

    fn engine_name(&self) -> PluginSystemKind;

    /// Post initialization, once the context is available. This means you should be able to
    /// run anything here that could modify the context before the main editor is available.
    fn run_initialization_script(&self, _cx: &mut Context) {}

    /// Allow the engine to directly handle a keymap event. This is some of the tightest integration
    /// with the engine, directly intercepting any keymap events. By default, this just delegates to the
    /// editors default keybindings.
    #[inline(always)]
    fn handle_keymap_event(
        &self,
        _editor: &mut ui::EditorView,
        _mode: Mode,
        _cxt: &mut Context,
        _event: KeyEvent,
    ) -> Option<KeymapResult> {
        None
    }

    /// This attempts to call a function in the engine with the name `name` using the args `args`. The context
    /// is available here. Returns a bool indicating whether the function exists or not.
    #[inline(always)]
    fn call_function_if_global_exists(
        &self,
        _cx: &mut Context,
        _name: &str,
        _args: &[Cow<str>],
    ) -> bool {
        false
    }

    /// This is explicitly for calling a function via the typed command interface, e.g. `:vsplit`. The context here
    /// that is available is more limited than the context available in `call_function_if_global_exists`. This also
    /// gives the ability to handle in progress commands with `PromptEvent`.
    #[inline(always)]
    fn call_typed_command_if_global_exists<'a>(
        &self,
        _cx: &mut compositor::Context,
        _input: &'a str,
        _parts: &'a [&'a str],
        _event: PromptEvent,
    ) -> bool {
        false
    }

    /// Given an identifier, extract the documentation from the engine.
    #[inline(always)]
    fn get_doc_for_identifier(&self, _ident: &str) -> Option<String> {
        None
    }

    /// Fuzzy match the input against the fuzzy matcher, used for handling completions on typed commands
    #[inline(always)]
    fn available_commands<'a>(&self) -> Vec<Cow<'a, str>> {
        Vec::new()
    }

    /// Retrieve a theme for a given name
    #[inline(always)]
    fn load_theme(&self, _name: &str) -> Option<Theme> {
        None
    }

    /// Retrieve the list of themes that exist within the runtime
    #[inline(always)]
    fn themes(&self) -> Option<Vec<String>> {
        None
    }

    /// Fetch the language configuration as monitored by the plugin system.
    ///
    /// For now - this maintains backwards compatibility with the existing toml configuration,
    /// and as such the toml error is exposed here.
    #[inline(always)]
    fn load_language_configuration(&self) -> Option<Result<Configuration, toml::de::Error>> {
        None
    }
}

If you can implement this, the engine should be able to be embedded within Helix. On top of that, I believe what I have allows the coexistence of multiple scripting engines, with a built in priority for resolving commands / configurations / etc.

As a result, Steel here is entirely optional and also remains completely backwards compatible with the existing toml configuration. Steel is just another layer on the existing configuration chain, and as such will be applied last. This applies to both the config.toml and the languages.toml. Keybindings can be defined via Steel as well, and these can be buffer specific, language specific, or global. Themes can also be defined from Steel code and enabled, although this is not as rigorously tested and is a relatively recent addition. Otherwise, I have been using this as my daily driver to develop for the last few months.

I opted for a two tiered approach, centered around a handful of design ideas that I'd like feedback on:

The first, there is a init.scm and a helix.scm file - the helix.scm module is where you define any commands that you would like to use at all. Any function exposed via that module is eligible to be used as a typed command or via a keybinding. For example:

;; helix.scm

(provide shell)

;;@doc
;; Specialized shell - also be able to override the existing definition, if possible.
(define (shell cx . args)
  ;; Replace the % with the current file
  (define expanded (map (lambda (x) (if (equal? x "%") (current-path cx) x)) args))
  (helix.run-shell-command cx expanded helix.PromptEvent::Validate))

This would then make the command :shell available, and it will just replace the % with the current file. The documentation listed in the @doc doc comment will also pop up explaining what the command does:

Once the helix.scm module is require'd - then the init.scm file is run. One thing to note is that the helix.scm module does not have direct access to a running helix context. It must act entirely stateless of anything related to the helix context object. Running init.scm gives access to a helix object, currently defined as *helix.cx*. This is something I'm not sure I particularly love, as it makes async function calls a bit odd - I think it might make more sense to make the helix context just a global inside of a module. This would also save the hassle that every function exposed has to accept a cx parameter - this ends up with a great deal of boilerplate that I don't love. Consider the following:

;;@doc
;; Create a file under wherever we are
(define (create-file cx)
  (when (currently-in-labelled-buffer? cx FILE-TREE)
    (define currently-selected (list-ref *file-tree* (helix.static.get-current-line-number cx)))
    (define prompt
      (if (is-dir? currently-selected)
          (string-append "New file: " currently-selected "/")
          (string-append "New file: "
                         (trim-end-matches currently-selected (file-name currently-selected)))))

    (helix-prompt!
     cx
     prompt
     (lambda (cx result)
       (define file-name (string-append (trim-start-matches prompt "New file: ") result))
       (temporarily-switch-focus cx
                                 (lambda (cx)
                                   (helix.vsplit-new cx '() helix.PromptEvent::Validate)
                                   (helix.open cx (list file-name) helix.PromptEvent::Validate)
                                   (helix.write cx (list file-name) helix.PromptEvent::Validate)
                                   (helix.quit cx '() helix.PromptEvent::Validate)))

       (enqueue-thread-local-callback cx refresh-file-tree)))))

Every function call to helix built ins requires passing in the cx object - I think just having them be able to reference the global behind the scenes would make this a bit ergonomic. The integration with the helix runtime would make sure whether that variable actually points to a legal context, since we pass this in via reference, so it is only alive for the duration of the call to the engine.

Async functions

Steel has support for async functions, and has successfully been integrated with the tokio runtime used within helix, however it requires constructing manually the callback function yourself, rather than elegantly being able to use something like await. More to come on this, since the eventual design will depend on the decision to use a local context variable vs a global one.

Built in functions

The basic built in functions are first all of the function that are typed and static - i.e. everything here:

• Typed
• Static

However, these functions don't return values so aren't particularly useful for anything but their side effects to the editor state. As a result, I've taken the liberty of defining functions as I've needed/wanted them. Some care will need to be decided what those functions actually exposed are.

Examples

Here are some examples of plugins that I have developed using Steel:

File tree

Source can be found here

filetree.webm

Recent file picker

Source can be found here

recent-files.webm

This persists your recent files between sessions.

Scheme indent

Since steel is a scheme, there is a relatively okay scheme indent mode that only applied on .scm files, which can be found here. The implementation requires a little love, but worked enough for me to use helix to write scheme code :smile:

Terminal emulator

I did manage to whip up a terminal emulator, however paused the development of it while focusing on other things. When I get it back into working shape, I will post a video of it here. I am not sure what the status is with respect to a built in terminal emulator, but the one I got working did not attempt to do complete emulation, but rather just maintained a shell to interact with non-interactively (e.g. don't try to launch helix in it, you'll have a bad time :smile: )

Steel as a choice for a language

I understand that there is skepticism around something like Steel, however I have been working diligently on improving it. My current projects include shoring up the documentation, and working on an LSP for it to make development easier - but I will do that in parallel with maintaining this PR. If Steel is not chosen and a different language is picked, in theory the API I've exposed should do the trick at least with matching the implementation behavior that I've outlined here.

Pure rust plugins

As part of this, I spent some time trying to expose a C ABI from helix to do rust to rust plugins directly in helix without a scripting engine, with little success. Steel supports loading dylibs over a stable abi (will link to documentation once I've written it). I used this to develop the proof of concept terminal emulator. So, you might not be a huge fan of scheme code, but in theory you can write mostly Rust and use Steel as glue if you'd like - you would just be limited to the abi compatible types.

System compatibility

I develop off of Linux and Mac - but have not tested on windows. I have access to a windows system, and will get around to testing on that when the time comes.

[zetashift]

I've been using this Steel + Helix integration for a bit. My config is here: https://github.com/zetashift/helix-config

The main downside here imho is that Helix isn't as great for Lisps(or Scheme in this case?) as Emacs for example.

But besides that this worked nicely. I have no real complaints and I actually did not need to script so many things as with my VSCode or NeoVim configs because a lot just works :tm: .

[pascalkuthe]

The main downside here imho is that Helix isn't as great for Lisps(or Scheme in this case?) as Emacs for example.

I think we can definitely improve, this is just the first prototype after all. You can't do everything at once. For example, I want to fully revamp the config system in the future (and remove the toml based config) so that is works even better with scheme

[zetashift]

I think we can definitely improve, this is just the first prototype after all. You can't do everything at once. For example, I want to fully revamp the config system in the future (and remove the toml based config) so that is works even better with scheme

Oh! I'm very sorry if I came over as demanding. I meant more like, in my experience this was what I stumbled upon, and it's not even a permanent thing, because a LSP for Steel might be in the works as well.

I completely agree that you can't do everything at once, nor should somebody be obliged to do it.

And maybe some lower hanging fruit can be picked up by people like me. For example if the tree-sitter grammar for scheme can improve things for Steel than that's another avenue.

[pascalkuthe]

No worries I didn't read it as negative. I ws more trying to highlight that this is just an initial prototype that can & will Improve (although Mathew already did a great job!)

[mattwparas]

And maybe some lower hanging fruit can be picked up by people like me. For example if the tree-sitter grammar for scheme can improve things for Steel than that's another avenue.

This is something that people with more familiarity with tree sitter and indents could be able to answer - are indent queries capable of matching a proper lisp indent mode? I did spend some time trying to mimic it but without much success

[Triton171]

This is something that people with more familiarity with tree sitter and indents could be able to answer - are indent queries capable of matching a proper lisp indent mode? I did spend some time trying to mimic it but without much success

I don't know Scheme / Lisp but it looks like the indentation should not be too difficult to model with tree-sitter indent queries (just using @indent and @align). I wrote a very simple query that correctly predicts the indentation of the first 100 lines of your scheme indent code:

(list) @indent
; Align lists to the second element (except if the list starts with `define`)
(list . (symbol) @first . (_) @anchor
  (#not-eq? @first "define")
  (#set! "scope" "tail")) @align

In order to improve this, I'd need to understand in which cases lists are aligned to the first, in which to the second element and when they are not aligned at all. Is this documented somewhere? If I find the time, I'll also try to understand the scheme indent implementation you wrote. Now that the plugin system is making progress, it might finally be time for me to learn the language :smile:

[mattwparas]

I don't know Scheme / Lisp but it looks like the indentation should not be too difficult to model with tree-sitter indent queries (just using @indent and @align). I wrote a very simple query that correctly predicts the indentation of the first 100 lines of your scheme indent code:

(list) @indent
; Align lists to the second element (except if the list starts with `define`)
(list . (symbol) @first . (_) @anchor
  (#not-eq? @first "define")
  (#set! "scope" "tail")) @align

In order to improve this, I'd need to understand in which cases lists are aligned to the first, in which to the second element and when they are not aligned at all. Is this documented somewhere? If I find the time, I'll also try to understand the scheme indent implementation you wrote. Now that the plugin system is making progress, it might finally be time for me to learn the language 😄

This is great, I really need to learn more about tree sitter idents! I used this as a reference to get started: https://github.com/ds26gte/scmindent#how-subforms-are-indented

Note, my implementation is not complete and was mostly an exercise in 1. seeing if it could be done and 2. getting something far enough to make editing scheme code more pleasant, I would not verbatim use it as a canonical reference.

[Triton171]

Thanks a lot for the reference @mattwparas. I created a PR (#8720) with indent queries that should cover everything from it except for some simplifications regarding keywords. Feel free to try it out and report any issues it has (you can just copy the indents.scm file into the corresponding folder in your runtime directory).

[nikolay]

Why Lisp, really?! Why not one of the WebAssembly languages? And why a Lisp dialect? Why bloat Helix further?

[pascalkuthe]

this PR is not intended to discuss the choice of plugin language. We already made our choice after lengthy discussion. This PR is only intended for reviewing and discussing this particular implementation. I will mark such comments offtopic

[kirawi]

As an aside question, I recall that we were talking about sandboxing on Matrix a while back. Is this still planned/implemented here?

[pascalkuthe]

As an aside question, I recall that we were talking about sandboxing on Matrix a while back. Is this still planned/implemented here?

Not fully, right now there are no restrictions what a VM van do but I talked about this a while ago with Mathew and it should be possible to add capability based security to plygins in the future.

This would entail a one VM per plugin approach (and how to handle interop in that case) but those should be solvable problems (the same problems would have occurred with any sandboxing including wasm)

[nalply]

I thought about a plugin system also using the PluginSystem trait proposed by @mattwparas: a plugin is a separate process communicating by an RPC channel (perhaps using the crate rpc-ipc).

Pros:

• The plugin can be developed in Rust
• If a plugin crashes, helix continues working
• Probably lightweight for Helix: only the RPC crate and glue code between PluginSystem and RPC
• With a second plugin system @mattwparas's trait can be tested harder

Cons:

• Definitively no sandboxing
• Community split between Scheme and Rust plugin developers
• RPC overhead
• has already been discussed and rejected

This is a quick idea which I posted here only because PluginSystem is described in this pull request.

[pascalkuthe]

while an advantage of hiding the plugin system behind a trait is definitely that the language can be swapped I don't think we will want to maintain too many alternatives in tree. Particularly not something RPC based which we have rejected often before (if it all something in rust then something shared library based)

[pascalkuthe]

All kinds rpc no matter the details is not a fit. Either way as I said above this PRis not intended for discussing language choice and only for reviewing this particularl implementation so I will mark these comments as off-topic

[lewiszlw]

Hi, what's the status of this pr? Really looking forward more progress on plugin system.

[mattwparas]

Hi, what's the status of this pr? Really looking forward more progress on plugin system.

I'm still working on it! I made some reasonably involved changes to avoid passing around a context to all the functions which needs some clean up. I've also made a relatively well functioning terminal emulator with the existing component API. Stay tuned!

[amano-kenji]

I haven't looked into the code. Did you avoid passing around a context with state monad?

When I wrote haskell, I used state monad to carry implicit global state.

[kirawi]

Would you consider this ready to play around with and write some plugins for, or is it still too early for that?

[mattwparas]

Would you consider this ready to play around with and write some plugins for, or is it still too early for that?

Certainly functional to use and write plugins with. I've been daily driving on it for a long time now. I'll be rebasing to the latest stuff later today, and will update the installation instructions.

If you decide to start using it, please don't expect any guarantees around anything. I'd expect the configuration API to change before its all said and done.

[noor-tg]

What is needed to integrate this with helix master ?

[mattwparas]

What is needed to integrate this with helix master ?

There were some configuration api changes (I think) that I was waiting for to land first. After that, just need to address the remaining comments and do some clean up.

I'm currently traveling for a few weeks but will be back working on this in early June.

[jonas-w]

Thank you for implementing this! I just tried it out, and I think this will need some better IO error handling, maybe you are already working it, but here are the quirks I found:

1. If you don't have a helix.scm and open helix, it looks like this:

This error popup is also a bit annoying as it moves with the cursor and actions like Escape or <C-c> require to be pressed twice, as the first press only closes the popup and then the next press does the wanted action (Exiting insert mode/commenting out code)

2. If you set $STEEL_HOME to a non-existent directory it will show an error too, even without having a helix.scm/init.scm file, and it does not even show the error from 1., only the following:

3. Not really related to helix, but I also encountered it while trying to set up your helix-config repository (using the instructions you provided in https://github.com/mattwparas/helix-config/issues/1):

When running cargo xtask install on the steel repository, it will say:

Successfully installed `cargo-steel-lib`
--- Installing all dylibs ---
Error: NotPresent
Error: NotPresent
Error: NotPresent
Error: NotPresent
Finished.

At first, I wasn't really sure what the problem could be, but I needed to set $STEEL_HOME to a directory, and create the directory myself, then these errors stopped, and it installed the dylibs (at least it tried, it all errored out, but not important for this issue here, I needed to create $STEEL_HOME/native directory myself).

In the end it seems like I can't get your config to work, I've also opened an issue there (https://github.com/mattwparas/helix-config/issues/2), will definitely try again some time!

[Suya1671]

How would setting up steel work on something like nixos? Since the dylib stuff seems interesting

[cgahr]

Steel defines a flake.nix file which exports steel as package.

Hence, if you use nix flakes, you can add the steel repo and just install it as a package. Or you run nix run mattwparas:steel (I think).

[sullyj3]

it'd be nix run github:mattwparas/steel

[godalming123]

Thanks for finally fixing helix's biggest issue, and giving us plugins, @mattwparas! There's still a couple of things that could do with improvements though. Here's a my list of issues that I have found within my early first-impressions from a users perspective (they are checked if the issue has been fixed):

• [ ] This might already be implemented, but I think at some point (possibly in a future PR) the picker will have to be made more customizable so that there is not duplication of UI code between helix, and the plugin ecosystem:

  • A custom function to generate pick previews from a selected item
  • A custom function to open the item that is selected in the picker

  This would be useful (for example) for a file browser that has the list of directories/files in the main column of the picker, and shows a preview of the content of that item in the right column. Selecting a directory would then open up that directory in the picker.

• [x] This might be an issue with steel itself, but helix just crashes when a .scm file that is being imported is empty

• [X] When an import is invalid (EG: (require "path/to/file/that/does/not/exist.scm")), the error is super vague (Error: Io: No such file or directory (os error 2)). I think that the error message could have the following info added to it:

  • File location
  • Line number
  • Line contents

[voidcontext]

I am trying to make things work using a minimal config, my helix.scm looks like this:

(require (prefix-in helix. "helix/commands.scm"))
(require (prefix-in helix.static. "helix/static.scm"))

(provide open-helix-scm)

;;@doc
;; Open the helix.scm file
(define (open-helix-scm)
  (helix.open helix.static.get-helix-scm-path))

When I try to run the :open-helix-scm command I am getting a type mismatch error:

What am I doing wrong?

[mattwparas]

I am trying to make things work using a minimal config, my helix.scm looks like this:

(require (prefix-in helix. "helix/commands.scm"))
(require (prefix-in helix.static. "helix/static.scm"))

(provide open-helix-scm)

;;@doc
;; Open the helix.scm file
(define (open-helix-scm)
  (helix.open helix.static.get-helix-scm-path))

When I try to run the :open-helix-scm command I am getting a type mismatch error:

What am I doing wrong?

(define (open-helix-scm)
  (helix.open (helix.static.get-helix-scm-path))) ;; <= Wrap this in parentheses - you're not calling the function 
                                                                            ;;  when you pass it in directly like this

[voidcontext]

(define (open-helix-scm)
  (helix.open (helix.static.get-helix-scm-path))) ;; <= Wrap this in parentheses - you're not calling the function 
                                                                            ;;  when you pass it in directly like this

🤦‍♂️ indeed, works like a charm! Thank you for all of your work, this is amazing! Finally I can extend helix with some much needed functionality, in a way it wasn't possible before! 🙏