Scriptability / Extension API

[christianparpart]

trafficstars

I really thought I'm having a ticket for this one already, well yeah.

Have the ability to load third-party extensions using some API. Ideally this could be WebAssembly (e.g. via https://github.com/wasmx/fizzy) to load extension modules.

Possible use-cases

• custom background: wallpaper
• custom background: video
• VT sequence stats generator
• custom VT sequences
• maybe even the grid segment folding could be implemented as extension? (Not sure yet)
• custom cursor rendering (e.g. animated, nyancat, rainbow, multi-columnl)
• animated cursor movements (e.g. sliding from one cell to another)
• foreground/background color altering before
• multi-user controlled terminal session (e.g. useful for remote pair programming or remote job-interview tasks on the terminal)
• terminal server mode - would require the extension to open up a (e.g.) TCP listener that clients can connect to and read from / write to, e.g. useful for web clients that want to access that instance (useful for multi-user controlled terminal, see above)
• emoji input (with nicely looking input dialogs?)
• use custom terminal app, e.g. not locally spawned process but an SSH/telnet client, ....
• https://github.com/microsoft/terminal/issues/4000 also has a nice list of use-case ideas.

API thoughts

• should have zero performance impact on API endpoints / callbacks that are not in use by any extension
• reading configuration
• hook into terminal widget redraw to inject custom background (e.g. wallpaper / video)
• request repaint in X milliseconds (needed for video playback)
• callback to all VT sequences to allow creating extensions that generate some stats about what VT sequences have been used
• ability to construct new VT sequences
• read the grid buffer, including scrollback and viewport, and other terminal state
• modifying grid buffer / terminal state.
• alter the RenderBuffer after population and before double buffer swap.
• access the local file system for read/write (via access control)
• access the network/internet (via access control)
• Load and execute custom pixel/fragment shaders

other thoughts

• Each extension could/should be able to create new threads, but a few callbacks should be run in renderer (main) thread, e.g. for paint actions, as well as terminal I/O thread, e.g. for VT sequence hooks.
• A crashing extension should not affect others nor the core.
• Extensions should be able to be loaded at startup as well as runtime and unloaded/reloaded at runtime.
• If WT happens to decide on WebAssembly, too, then we could attempt to expose the same WASM API to the extensions, to share the same set of extensions (should make it easier for both, users as well as extension devs)

[WSLUser]

If WT happens to decide on WebAssembly, too, then we could attempt to expose the same WASM API to the extensions, to share the same set of extensions (should make it easier for both, users as well as extension devs)

You'd probably want to get buy-off from ConEmu as well as the other leading terminal on Windows.

Based on other issues, WT isn't keen to add any web technology if it can help it. It's one thing for a terminal that's web-based to begin with, using WebAssembly makes perfect sense for those terminals as they're not exposing a new surface to finangle with, but in the case of WT and even ConEmu, they aren't using any web technologies and as result, don't have any of the associated security issues you see popping up all the time.

As an aside, if a web tech was to be adopted, I have a feeling it would be more along the lines of PWA, which Edge uses, and because Edge is x-plat, this may make it easier for you. I doubt it would be PWA itself but probably something similar, which may use WebAssembly in the backend, not entirely sure there, as it's intended to localize a web app to not require the web to use.

[christianparpart]

WebAssembly (WASM) has its roots in the Web, but the technology is platform agnostic. Look at MS Flight Simulator. All extensions are interfacing via WASM. Think of it as a JVM competitor. WASM is also just a standard on how a VM should act and interface with apps running on it. You can use WASM and still have zero dependencies to the web. I am eyeing at using github.com/wasmx/fizzy for that.

[WSLUser]

You might want to bring this up on WT's extension tracker and see if that's a route they'd be willing to go. The first issue you'd run into for them using Fizzy is the fact they prefer MIT license. However I see it's under Apache-2.0, which MS itself uses for some projects, so legally, they'd probably just have do some paperwork to accept it as a dependency, and wait for that process to finish before taking an sort of actionable on it. After that, it would be a matter of working with them to develop something that can work for both terminals and preferably in a way agnostic enough to allow another terminal like ConEmu (which is also written in C/C++) to adopt as well if they ever choose to support extensions.

[christianparpart]

It is not about fizzy. That is just what i would use. WASM is just a Standard.

[WSLUser]

Yep. But they're going to want to go with a solution that provides the most bang for the least buck and if fizzy helps with that (and has a similar implementation to Contour) they'll want to use it.

[data-man]

https://github.com/dbohdan/embedded-scripting-languages Personally, I'd prefer https://github.com/ChaiScript/ChaiScript, but the compilation is very slow. :(

Also https://github.com/Tencent/ScriptX seems interesting.

[christianparpart]

The problem is that these curve you into a syntax that people may or may not like. WASM as compilation target is completely free of choice as long as your language of choice asked you to target WASM. :)

[data-man]

Now I prefer Luau. :)