Pen/Stylus support

[daxpedda]

Waiting for #3833.

---

This implements Pen/Stylus support inspired by the Web Pointer API.

• Instead of adding new events I decided to add them to CursorMoved and MouseInput (now renamed to CursorInput.
• This does preclude any larger changes in the spirit of #336, but I'm happy to include some changes in this PR.
• Despite the Web API not giving erasers their own device type, but simply a button, I decided to add a new device type instead. See https://github.com/w3c/pointerevents/issues/134 for why this was considered a mistake to begin with.
• X11 was emitting pen input in DeviceEvent::MouseWheel, DeviceEvent::MouseMotion, WindowEvent::CursorMoved, WindowEvent::MouseInput, which I disabled, because users would think all these inputs are mouse. This is also in line with all the other backends, X11 and iOS were the only ones emitting any pen events.
• iOS was emitting Force::Calibrated::altitude_angle when a pen is used. I removed this and disabled pen events as well, as this information is now part of the ToolState, which can be used with Force::normalized() to get the old behavior back.

Open questions:

• Rename "Tool" to "Stylus": https://github.com/rust-windowing/winit/pull/3810#discussion_r1686318673.
• Rename "Cursor" to "Pointer": https://github.com/rust-windowing/winit/pull/3810#discussion_r1686465957.
• Change DeviceEvent::MouseMotion to DeviceEvent::CursorMotion?
• Implement X11 before merging? (I'm happy to do it)

Follow-up:

• More compatibility issues with browser: #2875.
  • Linux Firefox has no pen support (Bugzilla).
• #336
• Gesture API? This might be able to include things like #3768 and all the Apple specific gestures.
• Implement more backends.
• X11 improvements/platform consistency (maybe after #2875 and a touch overhaul)
  • Don't move mouse with touch (?): https://github.com/rust-windowing/winit/pull/3810#discussion_r1686380331.
  • Don't move mouse on focus (?): https://github.com/rust-windowing/winit/pull/3810#discussion_r1686820482.
  • Better device detection: https://github.com/rust-windowing/winit/pull/3810#discussion_r1686862301.
  • Use correct device ID (?): https://github.com/rust-windowing/winit/pull/3810#discussion_r1686864895.

Public API changes

pub enum WindowEvent {
    CursorMoved {
        device_id: DeviceId,
        position: PhysicalPosition<f64>,
        // new field
        r#type: CursorType,
    },
    // renamed from `MouseInput`
    CursorInput {
        device_id: DeviceId,
        state: ElementState,
        // changed type from `MouseButton`
        button: CursorButton
    },
    ...
}

pub enum Force {
    Calibrated {
        force: f64,
        max_possible_force: f64,
        // removed field
        // altitude_angle: Option<f64>,
    },
    Normalized(f64),
}

impl Force {
    // takes new `angle` parameter
    // now always emits normalized force, only emits perpendicular force if `angle` is passed
    pub fn normalized(&self, angle: Option<ToolAngle>) -> f64 { ... }
}

// new types from here on

pub enum CursorType {
    Mouse,
    Pen(ToolState),
    Eraser(ToolState),
}

pub struct ToolState {
    pub force: Force,
    pub tangential_force: Option<f32>,
    pub twist: Option<u16>,
    pub tilt: Option<ToolTilt>,
    pub angle: Option<ToolAngle>,
}

impl ToolState {
    pub fn tilt(self) -> Option<ToolTilt> { ... }
    pub fn angle(self) -> Option<ToolAngle> { ... }
}

pub struct ToolTilt { pub x: i8, pub y: i8 }

impl ToolTilt {
    pub fn angle(self) -> ToolAngle { ... }
}

pub struct ToolAngle { pub altitude: f64, pub azimuth: f64 }

impl Default for ToolAngle { ... }

impl ToolAngle {
    pub fn tilt(self) -> ToolTilt { ... }
}

pub enum CursorButton {
    Mouse(MouseButton),
    Pen(ToolButton),
    Eraser(ToolButton),
}

impl From<MouseButton> for CursorButton { ... }

pub enum ToolButton {
    Contact,
    Barrel,
    Other(u16),
}

impl From<ToolButton> for MouseButton { ... }

Fixes #99.

[daxpedda]

After some discussion and thinking, I created #3833, which would basically result in a pre-cursor to this PR. The implementation of #3833 will be partly based on the code here.

I will resume this PR when we are done with #3833.

[DorianRudolph]

Great work on this PR and the pointer event overhaul! May I ask what the status of this PR is? This is something I have wanted for quite some time.

I made an implementation based on this PR (with some modifications) for wayland https://github.com/WhiteboardCX/winit/tree/tablet

• I use f64 for tilt/twist/angle because wayland reports these as floats and it feels more natural.
• I only use a single Tool kind for PointerKind, instead of separate eraser and pen. This aligns more with the wayland interface. The buttons on the pen are handled as PointerButton events and mapping these to pen/eraser/... then can be handled by the application.
• I also added a ToolType, currently only pen and eraser, though wayland supports many more. However, I don't have hardware with different tool types. Presumably pens with an eraser on the back should work.
• Wayland provides some extra features like distance, hardware ids for pens, etc., which presumably are also only available on higher end wacom tablets (I only have Huion and "One by Wacom"). I did not implement these.
• Strangely, the wayland protocol does not seem to have a way to associate the tablet tools with the drawing tablet if there are multiple (not sure if it matters). The device_id is just assigned in the order in which the pens appear (my tablet don't report any of the ids in the protocol). So the same tool could get a different id when it reappears.

If my changes are OK, I can also make a PR. I should have hardware to test all platforms, but implementing all of them will take some time.

[madsmtm]

May I ask what the status of this PR is

It's nominated for our next meeting, which should happen sometime in start January

[DorianRudolph]

Thanks. In the meantime, I added support for windows to my fork and created a small demo based on vello.

[wareya]

Does this PR need to be redone on top of https://github.com/rust-windowing/winit/pull/3876? I have a lot of free time right now and like the above poster I did a test implementation of exposing tablet events on Windows. I could do the same for linux, android, and web too if this PR's creator is busy. https://github.com/wareya/winit/tree/tablet

[DorianRudolph]

Hey, I think my version of this was already on top of that commit https://github.com/WhiteboardCX/winit/tree/tablet I also had Linux/Wayland and windows support.

I would be happy to work with you on this. In principle, I also have Mac and iPad hardware, so I could try adding support for these as well.

At this point my fork probably needs some work again to run on top of the latest master branch, and it would be helpful if someone else went over my design decisions.

[madsmtm]

Thanks for the interest. I unfortunately don't think we're equipped to review such work at the moment (I plan to post something about that soon), would it be okay if I maybe got back to you in a few months?

[wareya]

Are you planning on doing work that would require us to rewrite whatever we come up with, like swapping out platform integrations? If not, I'm willing to do whatever you guys need to make reviewing easier (e.g. limiting its scope, or sticking to a basic design that won't need breakage to be extended later, limiting the platforms supported at first to ones that are unlikely to need changing, etc).

[DasLixou]

Don't want to promise anything but it seems that after the backends are now split in seperate crates, the awaited "refactor" is done? I might try to adopt this PR, make it work with the split and also try to integrate Apple Pencil support. (Still, nothing promised, but I'd like to know if now is the time and if it would be appreciated)