HaoboGu
Encoder, ScrollWheel, Mouse support related proposal
I have seen that "Encoder" support is on the roadmap.
In ZMK if you add a "rotary wheel" to the keyboard, they assume, it is realized with encoders (like EC11), which will send pulses to the microcontroller, so these could be processed as the user clicks for example up/down buttons.
My current goal is to implement an AS5600 magnetic rotary sensor based "rotary wheel" to my keyboard, which has 4096/revolution resolution and it is not able to send pulses, but have to be queried trough I2C. This does not work well with the above mentioned encoder abstraction, it is more like a mouse movement, or mouse scroll (which is my goal).
So I would propose to take care of such devices and implement something more abstract instead of encoders, which is able to serve all above mentioned use cases. So
- each rotary sensor should be able to describe its resolution/revolution,
- and their sensor event should report their relative movement value every time. So for example if you would like to use the rotary as volume pot, the amount or rotation needed should be independent of the used sensors resolution in a given config. Also it would be nice to be able to hook may things on the same wheel (for example depending on the active layer, modifiers, etc.): volume, screen brightness, mouse scrolling, backlight hue changes, or even mouse movement simulation.
So in my example I would read the sensor every 100ms, calculate the relative movement and report it as an "1D relative move" event to RMK. EC11 encoders would do the same but based on pulses. RMK should be able to transform this event for example into a vertical (or horizontal) mouse wheel HID report with correctly filled resolution multiplier, if that is in the user config or into a bunch of volume+ keypresses if that is in the user config.
A few years ago I also played with embedded Rust on Blue Pills (before Embassy), so maybe I can join the development too, but for that I'll need some initial guidance if you are interested...
Thanks for proposing it! I'm currently working on input devices framework now, in which a input device is abstracted as a device that sends Event:
pub enum Event {
Key(KeyEvent),
RotaryEncoder(RotaryEncoderEvent),
Touchpad(TouchpadEvent),
Joystick([AxisEvent; 3]),
... // to be extended
Custom([u8; 16])
}
pub struct RotaryEncoderEvent {
pub id: u8,
pub direction: Direction,
}
The whole system is fully event-driven, that means the input device(like rotary encoders) would wait for the next movement, then send out the event:
pub trait InputDevice {
async fn read_event(&mut self) -> Event;
}
The event processor would wait for the event from the input device and then process the event according to the event type.
At the first glance, it could cover your case(may need some modification on RotaryEncoderEvent). I would appreciate it if you could provide some feedbacks or suggestions :D
Yep, how nice, that Rust has sum-types! :)
Instead of RotaryEncoderEvent I would propose:
//for an EC11 encoder
pub struct RelativeMoveEvent {
pub id: u8,
pub value: i16, // -1 or +1
pub range: u16, // =15 (pulse per rotation, depends on the encoder)
}
//for my AS5600 sensor (can report absolute position, but I would convert and report it as relative)
pub struct RelativeMoveEvent {
pub id: u8,
pub value: i16, // anything in [-2048..2048) range
pub range: u16, // =4096 in case of AS5600
}
Digizer pads, joysticks may need absolute position event:
pub struct AbsolutePositionEvent {
pub id: u8,
pub value: i16, // anything in [0..range)
pub range: u16, // depends on the HW
}
Maybe a timestamp would be also useful, not sure...
For mouse movements (or for horizontal + vertical scroll wheels on a mouse) (or for joystic), the above could be combined into a multi dimensional array, but not sure that it is necessary:
pub enum Event {
Key(KeyEvent),
Absolute(AbsolutePositionEvent),
Absolute2D([AbsolutePositionEvent; 2]),
Absolute3D([AbsolutePositionEvent; 3]),
Relative(RelativeMoveEvent),
Relative2D([RelativeMoveEvent; 2]),
Relative3D([RelativeMoveEvent; 3]),
... // to be extended
Custom([u8; 16])
}
The "range" would be of course constant for each sensor type and could be stored elsewhere, but then I think itt would be difficult to reach for the information in the event handlers. Joysticks may also need calibration info, but that is not relevant from from the event's perspective and the input device "dirvers" can take care of it locally.
Then a separate mapping is needed, to take the data of the above listed raw sensor inputs and map it to keypress or other HID events, like:
pub enum HIDtargets {
InputAbsBrake,
InputAbsGas,
InputAbsMtSlot,
InputAbsRudder,
InputAbsRX,
InputAbsRY,
InputAbsRZ,
InputAbsThrottle,
InputAbsWheel,
InputAbsX,
InputAbsY,
InputAbsZ,
InputRelDial,
InputRelHorizontalWheel,
InputRelMisc,
InputRelRX,
InputRelRY,
InputRelRZ,
InputRelWheel,
InputRelX,
InputRelY,
InputRelZ,
}
During the mapping some transformation, scaling may also be needed (Input Processors in ZMK).
So if a user has a trackball both on the left and the right half of a split keyboard, then one of them may be mapped to work as scrollwheel, the other as mouse.
Take a look a ZMK documenation, look for "Input Processors", "behavior-sensor-rotate", "behavior-sensor-rotate-var", "behavior-input-two-axis". (In the code it seems that they can simulate the acceleration of mouse pointers too - of course that is more relevant if you want to drive the mouse with key presses.)
//for an EC11 encoder pub struct RelativeMoveEvent { pub id: u8, pub value: i16, // -1 or +1 pub range: u16, // =15 (pulse per rotation, depends on the encoder) } //for my AS5600 sensor (can report absulute position, but I would conver and report it as relative) pub struct RelativeMoveEvent { pub id: u8, pub value: i16, // anything in [-2048..2048) range pub range: u16, // =4096 in case of AS5600 }
If we do this, the event processor actually processes the same event differently according to the event's range?
I was assuming that 'id' is the sensor ID. So if he uses 3 identical rotary wheels and 2 identical trackballs, each instance would have different ids. So when the event processor working, it would look up every time what actions are bonded to that id.
This little refactor may make sense also:
pub struct RelativeMoveEvent {
pub value: i16, // -1 or +1
pub range: u16, // =15 (pulse per rotation, depends on the encoder)
}
pub struct AbsolutePositionEvent {
pub value: i16, // anything in [0..range)
pub minimum: i16, // depends on the HW
pub maximum: i16, // depends on the HW
}
pub enum EventData {
Key(KeyEvent),
Absolute(AbsolutePositionEvent),
Absolute2D([AbsolutePositionEvent; 2]),
Absolute3D([AbsolutePositionEvent; 3]),
Relative(RelativeMoveEvent),
Relative2D([RelativeMoveEvent; 2]),
Relative3D([RelativeMoveEvent; 3]),
BatteryStatus,
... // to be extended?
}
pub struct Event
{
pub id: u8, //event source id
//pub instant, //timestamp if needed
pub data: EventData,
}
I think in @HaoboGu 's design, some jobs of processing is done by the input_device(the driver). For instance, the input_device will send the mouse event when they are configured to act as a mouse, even if you want to act a key press behavior with the rotary device, it would be done by the driver as well.
As for the range, I think no matter what the resolutions of the devices are, it can map into a correct range of i8 or i16. If users' devices only support one direction, in the driver implement, it can map into the range of positive.
But as @tib888 mentioned, this design makes the interaction between different devices hard, like the layer states or some combos related to more than one device.
I was assuming that 'id' is the sensor ID.
ah yes, the id is the sensor id. In my initial design, the id is used for distinguishing different instances of the same device type, like multiple rotary encoder. The different types of input devices are recognized by the type of event they emit. This is because multiple event processors might be added to the framework, such as EncoderProcessor, KeyProcessor, etc. So we don't need id <-> device binding info any more. It improves the robustness of the system, but as @drindr said, it makes it a little difficult for sharing states across processors. @tib888 What do you think? I'm glad to hear thoughts on it, the push the design to a better direction.
Well I think it is not practical to reconfigure the drivers every time the user changes the layer or presses a modifier key.
The sensors should just report raw movements, then the attached behaviors should be able to change the report based on the layers, modifier keys, etc. (For example with a clever combination of behaviors a trackball could normally report mouse movement, but on a other layer report horizontal/vertical scroll or volume up / down if the shift is pressed (this later would require ZMK's mod-morph behavior)).
Also, if all raw events mapped in one type, I guess it is simpler to stream them from the splited parts to the central unit, which would do all the mapping and reporting to the PC. (Along this line I have added BatteryStatus to the event list - so every split part can report its battery status.)
Well I think it is not practical to reconfigure the drivers every time the user changes the layer or presses a modifier key.
I cannot agree with you anymore.🥰
I think the reason why it just does not use the raw movements is for some extensibility reasons. In the philosophy of RMK, it's a lib crate. The raw movements hard coding in the Event may make it difficult to add some highly-customed devices. So the Event is designed more closed to HID-end rather than the device-end. This is just my personal understanding.
I think besides the API currently in progress, a states sharing mechanism between devices is significant for some combination behaviors.
There might be some misunderstanding here..The device side won't be reconfigured, the processors are responsible for the state changes. The current device implementation are more like the "raw" physical event of device's, so I think it's more closer to the device side, not HID side. For example the rotary encoder reports only the movement direction, and the touchpad would report the relative movement or absolute position, plus the gesture(if gestures are support).
And for encoders, according to the type（ec11 or as5600）, they can emit both "directions" or "movement". These two types of the event would be processed by two different processors. That's what I mean.
If so, the states sharing mechanism between processor should be considered🤔
Definitely!
I'm considering to use &RefCell<KeyboardState> to share the states across the processors, just like the Keymap we have now. Mutex<RefCell<Option<KeyboardState>>> is also an option.
Ah I just realized, the AxisEvent is just like what @tib888 proposed(yeah it lacks range, which would be added later):
pub struct AxisEvent {
/// The axis event value type, relative or absolute
pub typ: AxisValType,
/// The axis name
pub axis: Axis,
/// Value of the axis event
pub value: i16,
}
pub enum AxisValType {
/// The axis value is relative
Rel,
/// The axis value is absolute
Abs,
}
#[non_exhaustive]
pub enum Axis {
X,
Y,
Z,
H,
V,
// .. More is allowed
}
The full (draft) implementation is at: https://github.com/HaoboGu/rmk/blob/main/rmk/src/event.rs
I have been inspired by this video: https://www.youtube.com/watch?v=FSy9G6bNuKA please take a look, and you will understand, why the resolution is important for me. :)
I haven't studied much the source code yet, since I'm just building my first keyboard. I looked around various firmwares starting with RMK since I love Rust. Unfortunately some features were missing, so I took a look at ZMK too, but it needed a LOT of documentation reading to be able to add my own code, difficult and slow to compile locally, and it is C, no type safety, full of macros, etc. and after days wasted on this, I am still not able to get my little code working in that device tree. So rather I came back here to join the development of missing features and maybe apply here some good concepts I learned there...
I'll deep dive into RMK source code this weekend and get back to this conversation.
If so, the states sharing mechanism between processor should be considered🤔
Definitely!
I'm considering to use
&RefCell<KeyboardState>to share the states across the processors, just like theKeymapwe have now.Mutex<RefCell<Option<KeyboardState>>>is also an option.
I agree: state sharing is necessarry if there will be more than one processors. In this case the various processors should "know eachother" well, and breaking changes in a state struct of one processor will affect all the dependent processors, so they can not be just independently developed "plugins".
The other possibility would be to have only one processor, which provides a framework for mapping from any supported event types, to any supported hid reports. I think this would need only a relatively small number of simple and configurable mapping behaviors (instead of the above case where each processsor's rust code have to be edited for customization).
I would vote for this later "only one processor" solution.
An example to build a graph of event flow from toml config: (ids are sensor ids, the output of many transformer act as a virtual sensor: sends events with its configured output id)
[event_flow]
#combine two independent axis sensor into a 2D one, and send a mouse HID report after a transformation
axis_combiner_2 = { first_axis1d_id = 1, second_axis1d_id = 2, out_axis2d_id = 5 }
axis2d_transformer = { axis2d_id = 5, matrix = { ... }, out_axis2d_id = 6 }
axis2d_to_hid = { axis2d_id = 6, hid_target = "InputRelMouseXY" }
#get the 1th coordinate of a 2D axis and if shift is pressed, report as vertical scroll
axis_splitter = { axis_id = 6, axis_index = 1, out_axis1d_id = 7 }
conditional_forward = { event_id = 7, when_modifiers = "LShift,RShift", output__id = 8 }
axis_to_hid = { axis1d_id = 8, hid_target = "InputRelWheel" }
#depending on the rotation direction of an axis or encoder sensor execute different key actions
axis_direction_to_keyaction = { axis1d_id = 3, positive_dir_key = "PgUp", negative_dir_key = "PgDown" }
#drive mouse (or joystick or whatever) with keypresses
keyaction_to_axis_hid = { key = "Left", movement = "5", hid_target = InputRelHWheel }
Of course better formats can be invented, but I hope, you get the idea. (To configure this of course vial is not ideal, rather search for "visual event graph editor" pictures.) (Even battery level reports could be used as input events...) (Even RGB backlight or display drivers could be hooked on this graph, which react to certain events... )
In this case the various processors should "know eachother" well, and breaking changes in a state struct of one processor will affect all the dependent processors, so they can not be just independently developed "plugins".
Yes, the state should be a global state, all processors share the newest state of the keyboard. I'm considering to add a ProcessorManager which manages all processors.
For the single processor way, I worry that it would takes more RAM/FLASH since it includes all functionalities into one processor instance. How it could be easy to extend is another thing that I have to concern.