microsoft
Input system overhaul
I am making this issue to discuss a possible way of implementing the next input system. Right now, input system binds the shortcuts directly in code, which makes it hard to change for the end user, as well as creates some complications when adopting the editor to platforms with unusual terminal behaviours (e.g. macOS).
How input system can be implemented?
Constraints
As was already discussed here, inputs can:
- Be declared outside of the main binary (see: extensions plan)
- Work on condition (see: menubar)
- Overlay each other (e.g. multiline textarea in modal and enter key) Additionally, following factors can influence the decision:
- No callbacks by design
- Immediate render mode So, what can be done? Solution that I came up with looks like this:
Overview
We already have two systems that will help us: Previous node tree and focus. What I want to do, is to insert into node tree "Action scope boundaries", that will have scopes associated with them. When the key is pressed, we are going to start at the node that is focused, and find the closest action boundary that has this shortcut/key bound for its scope, and fire the action associated with this shortcut. On the render pass, we can return the action only when we are in the same scope again. Code that is expected if this will be the chosen method:
fn setup(tui: &mut Tui) {
tui.declare_scope("menu-a", &[("back", vk::ESC)]);
}
fn render_menu(ctx: &mut Context, state: &mut State) {
ctx.scope_begin("menu-a", 12);
// Normal rendering logic here...
if ctx.consume_action("back") {
state.menu_open = false;
}
ctx.scope_end();
}
Pros and cons
Pros:
- Does not require any enums/hardcoding
- Reasonably easy to create keymap overrides
- Does not require callbacks, works reasonably well for immediate mode rendering
Cons:
- Complicates consumption of input
Let me expand on that.
Complications
Let's define two UI elements:
fn list_render(ctx: &mut Context, state: &mut State) {
ctx.scope_begin("list", 1);
for i in 1..11 {
checkbox_render(ctx, state, i);
}
ctx.scope_end();
}
fn checkbox_render(ctx: &mut Context, state: &mut State, n: usize) {
ctx.scope_begin("checkbox", n)
ctx.scope_end();
}
Let's assume that both "list" and "checkbox" scopes have DEL button bound. For checkbox, DEL deletes the checkbox sometimes (for now let's say 50% of the time), for list it deletes the list itself (always).
Let's consider this state (I will write it in xml, just for ease of understanding), including action boundaries (tag asb):
<asb scope="list" key=1>
<list>
<asb scope="checkbox" key=1>
<checkbox/>
</asb>
</list>
</asb>
Human pressed DEL, with the focus on the checkbox, and the checkbox have not been deleted. That means that now we need to propagate the action up, because it's not handled. But, for the system to give you back DEL bound to the "list" scope, it needs to either:
- Be sure that checkbox with key 1 did not handle the key
- checkbox with key 1 never appeared in the render sequence
First one is easy to verify, but No. 2 cannot be guaranteed unless we closed the scope. So the only solution it to only allow developers to put key consumption AFTER all the children were rendered, and panic! if someone tried to first consume the key and then the child scope we were targeting with focus appeared down the render chain.
When I thought of this, I reasoned that this will make hiding/deleting elements harder (sometimes).
Well, this is not only option. But the second option requires us to be able to abandon rerenders, which makes everything much more complicated and probably undesirable.
I am creating this issue to discuss this way of handling the input, because I don't want to make a large PR without getting any feedback on the idea
My previous suggestion was to pass the scope directly as an argument directly to consume_action, e.g.:
fn render_menu(ctx: &mut Context, state: &mut State) {
// Normal rendering logic here...
if ctx.consume_action("menu-a", "back") {
state.menu_open = false;
}
}
This may work well because a "button" for instance will almost always have a scope of "button" right? But your scope idea solves the issue of "What if that's not the case?", which I like. It also allows us to correlate what type of "input scope" the current focus path is in. This may be useful long-term to show help messages.
I'm not entirely sure if this is the final approach we should use, but I do think this is worth prototyping!
So the only solution it to only allow developers to put key consumption AFTER all the children were rendered, and panic! if someone tried to first consume the key and then the child scope we were targeting with focus appeared down the render chain.
We can model this after ImGui as well. For instance: https://github.com/ocornut/imgui/blob/77f1d3b317c400c34ee02fe9a5354d0d757b55ca/imgui.h#L980-L996
There's another solution to this though: Other immediate mode UI frameworks distribute input after building the UI tree, not during.
Basically, they build the UI tree as if a None input event was passed to our create_context. Then using the existing node tree (and node hashmap) the input event will be distributed from the root node down into the tree, following the current focus path.
The way consume_shortcut works is that you assign a "classname" (an ID) to the consume_shortcut calls as well, so they can be identified again on the next frame. Or you attach all consume_shortcut calls to a node. This then allows the framework to return true for the consume_shortcut call on the next frame.
This essentially goes to the same point about abandoning renders. Since we do not provide any separate update pass, render function is impure, and can be impure in many ways, including, if we are still planning the extensions, scheduling asynchronous IO. Because of this, we essentially have to treat all calls as impure, which heavily complicates the process. If we can assume that no input = no change, doesn't this mean that we can just use last render as the state just before the action gets consumed? And if it does, it still does not solve a dilemma that I presented with conditional propagation. I am not even talking about the layout, just element may sometimes decide that the key bind it received does not belong to it, resulting in propagation, which requires an ability to reexecute current render from the point in which we tracked parent consuming the thing (and abandon all changes children made) or to guarantee 100% that parents process their input only after all children did. Did I get something wrong here?
(Sorry, on my phone right now, text may be a little bit incoherent)
It's fine for the render process to be dirty and impure. The goal is not to write an actual UI framework after all, but rather to make our specific use-case of it work first and foremost. We don't have to ensure that parents consume input after children by design, since we can also just not... do that in our UI code. Not being a generic framework frees us from being bullet-proof.
Obviously, it would still be preferable to make it as easy to use and as robust as possible. But I believe this is secondary to making the system simple and first focus on making the editor itself feel great.
Lastly, immediate mode UIs are usually "approximate" (at least I don't know any that aren't). It's completely fine to take the previous frame state to decide whether to pass input to some consume_action in the next frame. This is already core to the current design (e.g. used for layout and for hit detection with the mouse). Due to this, I'm actually not sure if abandoning renders is necessary, but perhaps I'm misunderstanding it. (I was in a hurry when I read your issue, so I may have to re-read it later. 😅)
I think I understand what can be done. I will start implementing the prototype, hopefully will be able to complete it in 4 hours or so. Then I will create a draft PR and we will see what can be done.