zmk
zmk copied to clipboard
zmkfirmware
feat(mouse): Added mouse emulation
First of all, I'd like to thank @dtsykunov - this feature was our hackathon project.
Mouse button clicks, movement and scrolling from your keymap! There's even documentation available! This is still in alpha, though. Further development needs some architectural decisions, and some bugs need to be ironed out.
- Mouse movement over BLE is ~~very~~ (I've tried it, and it's actually usable. A MacOS tester on Discord reports not noticing a difference to USB) choppy. It's smooth at https://github.com/zmkfirmware/zmk/commit/bd108c5d9027bff194545d5c7fa1607fdfb2e24b, but it has frequent unpredictable deadlocks there, so it's unusable.
- The binding format needs to be decided on. Keypresses are fine, but the movement/scroll speeds would ideally be user-adjustable, and that requires wrangling signed/unsigned integers (which is tricky with DeviceTree) and deciding on the granularity (0-100 would lose a lot of the available precision).
- Currently, only mouse movement is handled via events. Ideally, all three behaviors would be, and the event structure needs to be decided on. Should all three mouse behaviours (click, movement, wheel) be consolidated into a single event and go straight into the report, or shoud there be three separate events that will go into the same report (easier to handle "hold-to-repeat" granularly)?
- On that point, try scrolling while holding a mouse movement button. It's even faster than an MX Master.
- The whole additive system with mouse movements and scrolling was basically thrown together in five minutes and should be re-examined. ~~I have a feeling it should be rewritted to
int16_ts and overflow checks should be implemented there.~~ The rewrite is done, and the usable value range is small enough that I don't think overflow checks are necessary. Still may need testing, though. - Mouse click code is also basically copied from modifiers. I'm pretty sure it's fine that way, but some parts may be excessive. Buttons 9-16 also don't work for some reason.
- Should movement press repeat timers be made user-configurable? I would imagine it'd be similar to mouse sensitivity adjustments.
- Some thought should also be given to future hardware support implementation. "hold-to-repeat" is not needed for actual mice, they also don't tend to "press" or "release" anything. By the way, if anyone can send a PS/2 trackpoint to Russia - we can work on integrating that too.
- Mouse actions should also be mappable to encoders.
I'll add to the list if I can think of anything else. In the meantime, testing and feedback would be very appreciated.
Any update on this feature?
Hi. This is being worked on, but Bluetooth stability proved to be a difficult hurdle to clear. There's an experimental branch with further improvements (acceleration, less crashing, etc.), you can find it in the ZMK Discord's #pointing-devices channel. That branch will eventually replace this one once the choppiness issue is resolved.
Just force-pushed the results of the last few months of work. Acceleration is fully implemented (at least as a first pass), and mouse movement over Bluetooth is much better, even usable by some reports. Unfortunately, I think we've gotten as far as possible with workqueues. They just don't work for time-sensitive applications such as these, and the movement is still not stable enough for my liking. The next step would be to rewrite the polling to use a dedicated thread, but that could be a while. In the meantime, testing and feedback on the acceleration and everything else would be more than welcome!
Hot to setup the pointer move speed, like KC_ACL0, KC_ACL1, KC_ACL2 in QMK?
If you look at the changes, there's a documentation bit about acceleration there. Here's a direct link: https://github.com/zmkfirmware/zmk/blob/621ecc54503851383736477360d6c8faac4d5cbe/docs/docs/behaviors/mouse-emulation.md#acceleration
Hot to setup the pointer move speed, like KC_ACL0, KC_ACL1, KC_ACL2 in QMK?
If you look at the changes, there's a documentation bit about acceleration there. Here's a direct link: https://github.com/zmkfirmware/zmk/blob/621ecc54503851383736477360d6c8faac4d5cbe/docs/docs/behaviors/mouse-emulation.md#acceleration
I am new to zmk. I wonder how I can change the acceleration config by pressing different acceleration keys. In QMK, I can press KC_ACL{0..2} keys to get different speed of the move and scroll.
&mmv {
time-to-max-speed-ms = <10>;
acceleration-exponent = <1>;
};
&mwh {
time-to-max-speed-ms = <10>;
acceleration-exponent = <1>;
};
/ {
The pointer move and scroll speed is too slow with above config. I tested different acceleration-exponent(0 to 2 even 1000), they all goes too slow for me.
Question:
- how I can gain faster speed
- if I can get different speed by just pressing some "acceleration key", like KC_ACL0, KC_ACL1, KC_ACL2 in QMK?
After reading this pr, I find a workaround for my needs:
I define some faster MOVE keys in the dt-bindgs file
#define MOVE_UP1 MOVE_VERT(-1400)
#define MOVE_DOWN1 MOVE_VERT(1400)
#define MOVE_LEFT1 MOVE_HOR(-1400)
#define MOVE_RIGHT1 MOVE_HOR(1400)
and use them in my .keymap file.
After reading this pr, I find a workaround for my needs:
I define some faster MOVE keys in the dt-bindgs file
#define MOVE_UP1 MOVE_VERT(-1400) #define MOVE_DOWN1 MOVE_VERT(1400) #define MOVE_LEFT1 MOVE_HOR(-1400) #define MOVE_RIGHT1 MOVE_HOR(1400)and use them in my .keymap file.
That's not really a workaround, it's the intended way of setting maximum speeds.
I think there should be a &mlt to function like <.
Currently I can not do something like:
< 1 LCLK
I put some faster movement in layer 1, and want to use this way to do some workaround of tap-dance, e.g. tap to send LCLK key, hold to access faster mouse movement key.
@krikun98
Seems there is a small bug, when I turn-off my right side of corne-wireless keyboard(nice!nano), the mouse movement becomes laggy.
After reading this pr, I find a workaround for my needs: I define some faster MOVE keys in the dt-bindgs file
#define MOVE_UP1 MOVE_VERT(-1400) #define MOVE_DOWN1 MOVE_VERT(1400) #define MOVE_LEFT1 MOVE_HOR(-1400) #define MOVE_RIGHT1 MOVE_HOR(1400)and use them in my .keymap file.
That's not really a workaround, it's the intended way of setting maximum speeds.
Maybe it should be put in the docs (but put defines possibly in the keymap file probably), as I also struggled how to do that. It was the first thing I wanted to change. First I tried looking for an option like max-speed next to time-to-max-speed-ms in the binding. Just my comment as newbie user of this feature
But I tested and works like a charm, a life saver for me. Using with mac-os, on a nice-nano v2, default speed was waay to small. And a lifesaver as I try to not get my hands away from the keyboard. The smoothness had issues only with general connection issues (the key presses had also issues), it's super smooth when I sorted it out by setting bt power with CONFIG_BT_CTLR_TX_PWR_PLUS_8=y
Hey!
Been using this on windows for one month and really happy with it, any status on merge to master soon?
Hey!
Been using this on windows for one month and really happy with it, any status on merge to master soon?
Thanks for the feedback! It's not really ready at the moment, I'm currently rewriting the emulation code for better wireless performance with the integration of pointing devices in mind.
Hey again!
Tried forking your stuff and merging the latest from zmk master branch hoping it would just work...doesn't seem to, unfortunately, really want the latest features (macros, capsword and more!), is there a chance you could update this fork with the latest?
Hey again!
Tried forking your stuff and merging the latest from zmk master branch hoping it would just work...doesn't seem to, unfortunately, really want the latest features (macros, capsword and more!), is there a chance you could update this fork with the latest?
Here you go: https://github.com/ftc/zmk/tree/mouse-ftc
I wanted the latest features with mouse keys as well and it seems @krikun98 is busy with his master's thesis right now. I rebased @krikun98's changes on the main zmk branch and merged in some HID fixes from @crides . Use at your own risk, I am pretty new to the code base so there may be issues. However, I am currently running it and haven't noticed any problems.
Is there anything blocking this from being merged as is and accepting the warts as they are?
Keeping this open but not merged makes it more difficult for anyone (like myself) who want's to contribute some pointer device code since they don't want to write my own incompatible implementation but are also weary of building on something that's not accepted.
adding on to @DelusionalLogic, I am willing to help with whatever. I was reading the code last week and it seems pretty straightforward. I've been using this pr for two months now and it works great for me.
Here is a proposal to drive this pull request to completion: Divide these changes into easier to review pull requests with someone taking charge of each one. Each pull request will have full functionality and may be reviewed independently.
I propose the following four tasks but feedback is very welcome. Each of these tasks should be something that can be completed, tested, and reviewed on its own. This is made easier since @krikun98 has done most of the work, I think we just need people to commit to owning each task and driving it though code review and testing.
- Right/middle/center click.
- Scroll up/down/left/right functionality.
- Mouse movement keys. (I think this one should contain the emulation code mentioned by @krikun98 earlier.)
- Documentation on using this feature.
The full change set is over 1300 lines so we may want to cut things down even more.
Each pull request may still depend on the changes from a previous pull request, but the development and review does not need to be blocked (If you search for "stacked pull requests" there are a fair number of articles on how to do this).
I can take one of these pull requests if there are other people willing to pick up the other three. I am also willing to help anyone working on the other two or review pull requests. Unfortunately, I don't personally have the time to commit to driving the whole thing.
@petejohanson (cc @krikun98): What are your thoughts on this? Does it sound reasonable?
@DelusionalLogic , @gcgbarbosa, or anyone else: would you be willing to pick up one of the four tasks I listed?
Hi @ftc, I can definitely work on the documentation...
I don't think I'd have time to do 3, but 1 and 2 are also ok with me...
I have not found any bugs testing this PR
@mrlinuxfish I have not either so if you and the other reviewers decide to just merge it, I have no objections. However, you may want to use my rebased version. There is not a ton of work on top of what is already here, but I did fix a number of merge conflicts and update the HID code to be compatible with the new version of Zephyr.
https://github.com/ftc/zmk/tree/mouse-ftc
Please let me know if you would like me to open a pull request from my branch.
Here is a proposal to drive this pull request to completion: Divide these changes into easier to review pull requests with someone taking charge of each one. Each pull request will have full functionality and may be reviewed independently.
Give the scope of this, I definitely think this is warranted.
In particular, I know that for #2 and #3, @krikun98 had mentioned wanting to improve the timing/acceleration system to be smoother, so keeping this queued while we get some core in with item 1 would be good.
I propose the following four tasks but feedback is very welcome. Each of these tasks should be something that can be completed, tested, and reviewed on its own. This is made easier since @krikun98 has done most of the work, I think we just need people to commit to owning each task and driving it though code review and testing.
Agreed.
1. Right/middle/center click.
To be clear, I'd expect this to include some of the necessary core HID changes as well.
2. Scroll up/down/left/right functionality. 3. Mouse movement keys. (I think this one should contain the emulation code mentioned by @krikun98 earlier.)
I believe both 2 and 3 will likely need some shared acceleration logic, we might want to have that be in a "Step 1b" piece independently.
4. Documentation on using this feature.The full change set is over 1300 lines so we may want to cut things down even more.
Each pull request may still depend on the changes from a previous pull request, but the development and review does not need to be blocked (If you search for "stacked pull requests" there are a fair number of articles on how to do this).
Indeed.
I can take one of these pull requests if there are other people willing to pick up the other three. I am also willing to help anyone working on the other two or review pull requests. Unfortunately, I don't personally have the time to commit to driving the whole thing.
@petejohanson (cc @krikun98): What are your thoughts on this? Does it sound reasonable?
Very reasonable. I would love for @krikun98 to weigh in, if possible, since this is definitely his baby.
I would love for @krikun98 to weigh in, if possible, since this is definitely his baby.
I totally agree here I want to make sure what I am proposing is aligned with @krikun98 's wishes. Hopefully my suggestion gives @krikun98 more leeway to work on the parts that excite him and offload some of the more boring tasks to the rest of us.
I believe both 2 and 3 will likely need some shared acceleration logic, we might want to have that be in a "Step 1b" piece independently.
Factoring out an acceleration pull request seems like a really good idea. We could make the task for pull request 1b to simply build a reasonable interface around the existing accelerator. A corresponding testable feature could be accelerating key repeat. It may not have a lot of use cases but it could be handy for the arrow keys.
In particular, I know that for #2 and #3, @krikun98 had mentioned wanting to improve the timing/acceleration system to be smoother, so keeping this queued while we get some core in with item 1 would be good.
My hope is that factoring out an interface for the accelerator could also be helpful here. I think getting exactly the right acceleration is going to be an ongoing task. A clean interface between the components could further unblock progress towards making this a main branch feature. The next improvement on acceleration could be pull request number 5 as long as @krikun98 is on board with the interface changes.
To be clear, I'd expect this [pull request 1] to include some of the necessary core HID changes as well.
I think that is totally reasonable. This HID code also seems to be used by some of the other trackpad and trackball work. I am happy to take the first task with the HID stuff since I had to figure that out to rebase the pull request anyway.
I went ahead and factored out the mouse button stuff from the mouse move/scroll stuff. I hope that will help drive the review process forwards.
FYI, the PR seems to work well with the upcoming Zephyr 3.2 upgrade. Starting with FTC's rebase, all it needs is resolving a few minor merge conflicts and updating the headers to the new include paths.
I am keeping a branch around with those changes at https://github.com/urob/zmk/tree/mouse-3.2. Currently it merges smoothly into Pete's upgrade branch for https://github.com/zmkfirmware/zmk/pull/1499 (petejohanson:zephyr/3.1-upgrade). I am trying to keep up with Pete's force-pushes but in case anyone wants to try and it doesn't merge smoothly let me know and I'll push another rebase.
NB: I also have a experimental main-3.2 branch with both mouse and the Zephyr-3.2 upgrade merged into main (as well as a few other PR's) at https://github.com/urob/zmk/tree/main-3.2
It seems like the mouse keys stopped working on macOS on Bluetooth.
It’s still working over usb.
At least one other user is experiencing it here: https://github.com/urob/zmk-config/issues/2
The old solution of unpairing and re-pairing doesn’t seem to work.
I am also having this issue using an old branch that used to work.
So I think this might be caused by a macOS update rather than a problem with this PR.
Is anyone else experiencing this problem and does anyone have a solution?
Quick correction: The mouse PR on urob's fork is working fien with macOS Ventura 13.5.
My issue was caused, because I had another PR that added BT characteristics on top of this PR in app/src/hog.c.
If someone else encounters similar issues, you can check this discussion on the zmk discord: https://discord.com/channels/719497620560543766/845285481888743434/1151183774235762818