Feature request: mouse any-event Tracking (1003)

[AutumnMeowMeow]

Reference: https://www.invisible-island.net/xterm/ctlseqs/ctlseqs.html#h3-Any-event-tracking

This would round out mouse support, permitting 1) applications to know the mouse position always, and 2) implement focus-follows-mouse for panes.

[imsnif]

I really like the "focus follows mouse" part, though I think we'll have to give some thoughts about the UX here. I'm not sure it should be on by default, at least... I guess we'll have to play around with it and see how it feels.

Another point that I'm pretty certain about: I think it would be best to only send mouse movements to the focused pane (assuming of course the movements are over said pane).

But regardless - I'd very much love to see this feature in Zellij. I guess we'll also have to place ourselves in any-event tracking mode. I wonder how wide-spread support for this is in terminals?

If you're interested in working on this @AutumnMeowMeow - I'd be happy to point you to the relevant parts of the code-base to get what you need (if you want me to, ofc).

[AutumnMeowMeow]

Another point that I'm pretty certain about: I think it would be best to only send mouse movements to the focused pane (assuming of course the movements are over said pane).

I believe the most common implementation for terminals and embedded terminal panes/widgets is:

• No button down - mouse motion events are only passed to the terminal if the mouse is over that pane.
  • Looking at a few terminals: xterm, mlterm, contour will report events even if the window is not focused; wezterm only reports if the window is focused. I think it would make sense to either follow xterm's example and always send any-event if the mouse is over a pane whether focused or not; or make it configurable with whatever default you want.
• Button down - mouse motion events are passed to the focused terminal, with the coordinates clamped to the terminal dimensions. (So no negative coordinates when click-dragging off screen.) This is the click-and-drag scenario.

I guess we'll also have to place ourselves in any-event tracking mode.

zellij can activate both button-event and any-event in that order on startup, like: "\x1b[?1002;1003h", and then zellij will get any-event if the host supports that, and button-event if the host doesn't support any-event but does support button-event. Easy graceful fallback. (And a good test for terminals that don't loop on all the DECSET parameters. :-) )

I wonder how wide-spread support for this is in terminals?

It's pretty common. All of the terminals listed here with "yes" in the "Mouse Cursor" column support Any-Event, including screen and tmux when their host console supports it. (I believe konsole does now too, and maybe DomTerm -- I haven't updated that page in some time.)

I would be interested in doing this one. My mental model so far is:

• zellij-utils/src/input/event/mouse.rs: add a Motion(Position) to MouseEvent . Follow the compiler messages to all the cases.
• zellij-client/src/os_input_output.rs: modify ENABLE_MOUSE_SUPPORT / DISABLE_MOUSE_SUPPORT to include 1003.
• zellij-server/src/tab/unit/tab_integration_tests.rs: add sgr_mouse_mode_any_event tests
• zellij-client/src/input_handler.rs add MouseEvent::Motion case to handle_mouse_event
• zellij-server-src/pane/terminal_pane.rs: add fn mouse_motion()
• zellij-server-src/pane/grid.rs: add AnyEventTracking to MouseTracking. Follow the compiler messages to all the cases.
• zellij-server-src/pane/grid.rs: add mouse_motion_signal. Looks like I also need to modify all of the click signals to send any-event appropriately too when hold is true and tracking is any-event. (It seems like Hold could be removed in favor of Motion, as a simplifying refactor. But initially I would leave it.)

I have no idea on:

• Where the actual decoding of the host terminal stream into mouse events occurs.
• If any other widgets/plugins/etc. need to be modified to handle a new mouse event.

[imsnif]

I believe the most common implementation for terminals and embedded terminal panes/widgets is:

* No button down - mouse motion events are only passed to the terminal if the mouse is over that pane.
  
  * Looking at a few terminals: xterm, mlterm, contour will report events even if the window is not focused; wezterm only reports if the window is focused.  I think it would make sense to either follow xterm's example and always send any-event if the mouse is over a pane whether focused or not; or make it configurable with whatever default you want.

* Button down - mouse motion events are passed to the focused terminal, with the coordinates clamped to the terminal dimensions.  (So no negative coordinates when click-dragging off screen.)  This is the click-and-drag scenario.

I just have this feeling that with multiple panes open, having the cursor change the internal cursor location of panes I happen to pass over when trying to focus on a pane beyond them might be a little annoying. I'm definitely on board with having this configurable. Let's try out both options and see what we feel works better as a default?

zellij can activate both button-event and any-event in that order on startup, like: "\x1b[?1002;1003h", and then zellij will get any-event if the host supports that, and button-event if the host doesn't support any-event but does support button-event. Easy graceful fallback. (And a good test for terminals that don't loop on all the DECSET parameters. :-) )

:D

I would be interested in doing this one. My mental model so far is:

* zellij-utils/src/input/event/mouse.rs: add a Motion(Position) to MouseEvent .  Follow the compiler messages to all the cases.

* zellij-client/src/os_input_output.rs: modify ENABLE_MOUSE_SUPPORT / DISABLE_MOUSE_SUPPORT to include 1003.

* zellij-server/src/tab/unit/tab_integration_tests.rs: add sgr_mouse_mode_any_event tests

* zellij-client/src/input_handler.rs add MouseEvent::Motion case to handle_mouse_event

* zellij-server-src/pane/terminal_pane.rs: add fn mouse_motion()

* zellij-server-src/pane/grid.rs: add AnyEventTracking to MouseTracking.  Follow the compiler messages to all the cases.

* zellij-server-src/pane/grid.rs: add mouse_motion_signal.  Looks like I also need to modify all of the click signals to send any-event appropriately too when hold is true and tracking is any-event.  (It seems like Hold could be removed in favor of Motion, as a simplifying refactor.  But initially I would leave it.)

This all seems correct. Also @tlinford rightfully mentioned that we can probably do away with a lot of boilerplate by just passing a MouseEvent around instead of having separate methods of doing it. But we can leave things as they are for now I think.

I have no idea on:

* Where the actual decoding of the host terminal stream into mouse events occurs.

We use Wezterm's input parser here: https://github.com/zellij-org/zellij/blob/main/zellij-client/src/stdin_handler.rs#L47 Here we convert from the above lib's event to our own: https://github.com/zellij-org/zellij/blob/main/zellij-utils/src/input/mouse.rs#L22 - this happens here: https://github.com/zellij-org/zellij/blob/main/zellij-client/src/input_handler.rs#L96

* If any other widgets/plugins/etc. need to be modified to handle a new mouse event.

For now I don't think so. It would be cool to add this as a plugin capability, but I'm overhauling the plugin system soon, so I'd rather wait until that's done with.

[AutumnMeowMeow]

A question to think about for after any-event is in: Will zellij later support an optional floating text mouse cursor? Like this?

https://user-images.githubusercontent.com/99296476/185799572-70b98db6-a159-4618-8f75-6de93b3be199.mp4

In the video above, we have a demo running inside its own terminal window, analogous to how zellij-inside-zellij in a floating window might go. Since both outside demo and demo-inside-terminal are in any-event mode, and each inverts the cell where the mouse is, then when the mouse goes over the inside terminal with being double-inverted the mouse could disappear entirely.

My solution is for an "inner" window to notify the outside world that it will be drawing a text cursor, so the outside shouldn't draw one, via a privacy message:

ESC ^ hideMousePointer ESC \

ESC ^ showMousePointer ESC \

You can see both the implementation and the flaw: when not over the inner terminal, there are two mouse cursors.

Should zellij later on do floating mouse cursor, let me know if you do a different solution for this. zellij has a much larger user base, so I would change mine to match zellij.

[imsnif]

Sure @AutumnMeowMeow. IMO this is not a piece of UI I'd like to have - but we can keep this in mind if this changes.

[jondo2010]

@AutumnMeowMeow any progress on this? I'm forever missing pane-resizing with the mouse #1262 and would need this to impl I believe.

[AutumnMeowMeow]

@AutumnMeowMeow any progress on this? I'm forever missing pane-resizing with the mouse #1262 and would need this to impl I believe.

@jondo2010

Sorry, no progress to report. I have been mostly unable to code for the last year (again). :(

[imsnif]

I hope you are well @AutumnMeowMeow !

[tmpm697]

sr, but would love to able to resize pane with mouse :)

with only this addition is what i really needed.