ESP32-BLE-Mouse
ESP32-BLE-Mouse copied to clipboard
T-vK
feature request: Absolute mode
Is there any possibility that there will be an absolute mode? That would be more useful and intuitive since otherwise I don't see a way around resetting the mouse position if it has accidentally 'run against a border', if that makes sense!
Also, sorry, I'm new to this field
Maybe, but for this to work, you manually have to tell your ESP the screen resolution, screen orientation, cursor speed and cursor acceleration of the device you want to connect to and it also needs to know the current cursor position which can always change if you have a physical mouse connected as well or alternatively you can reset the mouse position (for example by moving it to the top left corner) before every move with expected side-effects and performance loss.
Something else that is being worked on is emulating touchscreen input: https://github.com/T-vK/ESP32-BLE-Mouse/issues/5
Ok, so from this I take you assume there is no feasible way to get parameters like screen resolution from any accessible HID library, I was kind of hoping that would be the case, since somehow logitech products seem to be able to do this (air mice). I guess I was hoping to not need to calculate XY (using accel settings etc) but to get it sent back over the BLE interface from the computer. Sad but the reset of the position will work as a workaround :) thanks
I highly doubt there is a way to do this without additional software running on the target device. I could be wrong, but Logitech most likely uses a custom driver or an additional piece of software that you have to install on the target device in order for this to work. This software could be communicating with the mouse using a rfcomm serial connection. I think there has been a proof of concept on having a Bluetooth serial connection and a BLE server running at the same time: https://github.com/beegee-tokyo/ESP32-Weatherstation/blob/93079caa319eaeb1c237782bf348cafd96e6e3c1/docs/bt_serial.cpp
@street-grease-coder : thanks for your interest on this issue and we did several tests enabling the absolute mode in HID Descriptor but neither ios14+ ("ble touchsreen" HID seems to be disabled since ios13.4 https://discussions.apple.com/thread/251720560) nor android8+ devices. It seems that using absolute HID Descriptor + uint16_t coordinates datatypes from working usb HID Descriptor is not enough and they might be an issue with the ble packets sent to the HID driver of the host ? Some product (like http://www.acrosscenter.com/) seems to have remote mouse control over (ble + app/driver). What is your target devices/os ?
@T-vK : thanks for your reply and surely having (spp + ble) at the same time could help sending from the host some calibration information such as the resolution and orientation. Could you explain where to indicate the screen orientation and resolution in the sp32 code ? I could try to modify these part to adjust some settings on the ble mouse.
Could you explain where to indicate the screen orientation and resolution in the sp32 code ? I could try to modify these part to adjust some settings on the ble mouse.
No, the code has not been written yet. ... Thinking about how this would be implemented, I think it would only be necessary to know the cursor speed and acceleration.
But I'd imagine it to look somewhat like this:
#include <math.h> // required because of fmod
void moveTo(float newX, float newY, float speed = 1, float acceleration = 0, bool isLandscape = true, float initialX = 0, float initialY = 0) {
if (!isLandscape) { // Swap x and y in portrait mode
float tmp;
tmp = newX;
newX = newY;
newY = tmp;
tmp = initialX;
initialX = initialY;
initialY = tmp;
}
float currentX = initialX;
float currentY = initialY;
float maxStepSizeX = (initialX < newX) ? 127 : -128;
float maxStepSizeY = (initialY < newY) ? 127 : -128;
// We can only move the mouse by up to +127 units or -128 units at a time
while ((int)currentX != (int)newX || (int)currentY != (int)newY) {
float stepValueX;
if (currentX == newX) {
stepValueX = 0;
} else if ((maxStepSizeX >= 0 && currentX+maxStepSizeX*speed > newX) || (maxStepSizeX < 0 && currentX+maxStepSizeX*speed < newX)) {
stepValueX = fmod(newX-initialX, maxStepSizeX*speed) / speed;
} else {
stepValueX = maxStepSizeX;
}
currentX += stepValueX*speed;
float stepValueY;
if (currentY == newY) {
stepValueY = 0;
} else if ((maxStepSizeY >= 0 && currentY+maxStepSizeY*speed > newY) || (maxStepSizeY < 0 && currentY+maxStepSizeY*speed < newY)) {
stepValueY = fmod(newY-initialY, maxStepSizeY*speed) / speed;
} else {
stepValueY = maxStepSizeY;
}
currentY += stepValueY*speed;
bleMouse.move(stepValueX, stepValueY);
}
}
This should be able to compensate for speed. I have no clue how to compensate for acceleration though. I don't even know how that is implemented on the OS side. It might even be implemented differently on different operating systems. Some may for example reset the acceleration to 0 once the cursor hasn't been moved for more than 2 milliseconds, others may give it 3 milliseconds and then some may slowly decrease the acceleration instead of doing it instantly... I don't know, it's very complicated and not like something that I want to deal with. Personally I disable cursor acceleration on all of my computers anyway.
Maybe this would be of help to anyone who actually wants to implement this: http://archive.is/20120907165307/msdn.microsoft.com/en-us/windows/hardware/gg463319.aspx
Have fun adjusting this to your needs:
/**
* Absolute mouse example
*/
#include <BleMouse.h>
#include <math.h> // required because of fmod
BleMouse bleMouse;
// Acceleration is not implemented
void moveTo(float newX, float newY, float speed = 1, float acceleration = 0, bool isLandscape = true, float initialX = 0, float initialY = 0) {
if (!isLandscape) { // swap x and y in portrait mode
float tmp;
tmp = newX;
newX = newY;
newY = tmp;
tmp = initialX;
initialX = initialY;
initialY = tmp;
}
float currentX = initialX;
float currentY = initialY;
float maxStepSizeX = (initialX < newX) ? 127 : -128;
float maxStepSizeY = (initialY < newY) ? 127 : -128;
// we can only move the mouse by up to +127 units or -128 units at a time
while ((int)currentX != (int)newX || (int)currentY != (int)newY) {
float stepValueX;
if (currentX == newX) {
stepValueX = 0;
} else if ((maxStepSizeX >= 0 && currentX+maxStepSizeX*speed > newX) || (maxStepSizeX < 0 && currentX+maxStepSizeX*speed < newX)) {
stepValueX = fmod(newX-initialX, maxStepSizeX*speed) / speed;
} else {
stepValueX = maxStepSizeX;
}
currentX += stepValueX*speed;
float stepValueY;
if (currentY == newY) {
stepValueY = 0;
} else if ((maxStepSizeY >= 0 && currentY+maxStepSizeY*speed > newY) || (maxStepSizeY < 0 && currentY+maxStepSizeY*speed < newY)) {
stepValueY = fmod(newY-initialY, maxStepSizeY*speed) / speed;
} else {
stepValueY = maxStepSizeY;
}
currentY += stepValueY*speed;
bleMouse.move((char)stepValueX, (char)stepValueY);
Serial.print((int)stepValueX);
Serial.print("x");
Serial.println((int)stepValueY);
Serial.println("Moved!");
}
}
// Only works if the target device has a cursor speed of 1x and no acceleration
// and resets the cursor position to 0|0 before every move.
void simpleMoveTo(float x, float y) {
moveTo(0,0,1,0,true,10000,10000); // Move mouse to position 0|0
moveTo(x,x,1,0,true,0,0); // Move mouse to position x|y
}
void setup() {
Serial.begin(115200);
Serial.println("Starting BLE work!");
bleMouse.begin();
}
void loop() {
if(bleMouse.isConnected()) {
// Example 1:
int screenWidth = 1920-1; // -1 because the first pixel is 0
int screenHeight = 1080-1; // -1 because the first pixel is 0
int cursorSpeed = 1;
int cursorAcceleration = 0;
int landscape = true;
int newX = 0;
int newY = 0;
int oldX = screenWidth;
int oldY = screenHeight;
Serial.println("Moving mouse to the top left corner");
moveTo(newX,newY,cursorSpeed,cursorAcceleration,landscape,oldX,oldY);
delay(1000);
Serial.println("Moving mouse to the bottom right corner");
newX = screenWidth;
newY = screenHeight;
oldX = 0;
oldY = 0;
moveTo(newX,newY,cursorSpeed,cursorAcceleration,landscape,oldX,oldY);
delay(1000);
// Example 2 (doesn't require screen resolution and assumes a mouse speed of 1x):
simpleMoveTo(500,500); // Move cursor to position 500|500
simpleMoveTo(1000,10); // Move cursor to position 1000|10
}
}
But as I said it does not compensate for acceleration!
@T-vK : thanks a lot for this absolute example and I have been testing it with Android 8.0 on Huawey MediaPad T5. Below some adjustments on your code to make it work :
- negative movement down to -127 (not -128 which overflows the range)
- simpleMoveTo was calling x twice
- on my case I need a 1.5 correction factor between the target coordinates (x, y) and the actual position on the tablet screen. Speed-wise it is quite fast and I will run some more tests to see if there is some glitches on the final movement. Accuracy is quite good as you use float for coordinates. Available to keep adjusting this example and make tests on more devices. Cheers.
/**
* Absolute mouse example
*/
#include <BleMouse.h>
#include <math.h> // required because of fmod
BleMouse bleMouse;
// Acceleration is not implemented
void moveTo(float newX, float newY, float speed = 1, float acceleration = 0, bool isLandscape = true, float initialX = 0, float initialY = 0) {
if (!isLandscape) { // swap x and y in portrait mode
float tmp;
tmp = newX;
newX = newY;
newY = tmp;
tmp = initialX;
initialX = initialY;
initialY = tmp;
}
float currentX = initialX;
float currentY = initialY;
float maxStepSizeX = (initialX < newX) ? 127 : -127;
float maxStepSizeY = (initialY < newY) ? 127 : -127;
// we can only move the mouse by up to +127 units or -128 units at a time
while ((int)currentX != (int)newX || (int)currentY != (int)newY) {
float stepValueX;
if (currentX == newX) {
stepValueX = 0;
} else if ((maxStepSizeX >= 0 && currentX+maxStepSizeX*speed > newX) || (maxStepSizeX < 0 && currentX+maxStepSizeX*speed < newX)) {
stepValueX = fmod(newX-initialX, maxStepSizeX*speed) / speed;
} else {
stepValueX = maxStepSizeX;
}
currentX += stepValueX*speed;
float stepValueY;
if (currentY == newY) {
stepValueY = 0;
} else if ((maxStepSizeY >= 0 && currentY+maxStepSizeY*speed > newY) || (maxStepSizeY < 0 && currentY+maxStepSizeY*speed < newY)) {
stepValueY = fmod(newY-initialY, maxStepSizeY*speed) / speed;
} else {
stepValueY = maxStepSizeY;
}
currentY += stepValueY*speed;
bleMouse.move((char)stepValueX, (char)stepValueY);
Serial.print((int)stepValueX);
Serial.print("x");
Serial.println((int)stepValueY);
Serial.println("Moved!");
}
}
// Only works if the target device has a cursor speed of 1x and no acceleration
// and resets the cursor position to 0|0 before every move.
void simpleMoveTo(float x, float y) {
moveTo(0,0,1,0,true,10000,10000); // Move mouse to position 0|0
moveTo(x,y,1,0,true,0,0); // Move mouse to position x|y
}
void setup() {
Serial.begin(115200);
Serial.println("Starting BLE work!");
bleMouse.begin();
}
void loop() {
if(bleMouse.isConnected()) {
// Example 1:
int screenWidth = 1920-1; // -1 because the first pixel is 0
int screenHeight = 1080-1; // -1 because the first pixel is 0
int cursorSpeed = 1;
int cursorAcceleration = 0;
int landscape = true;
int newX = 0;
int newY = 0;
int oldX = screenWidth;
int oldY = screenHeight;
Serial.println("Moving mouse to the top left corner");
moveTo(newX,newY,cursorSpeed,cursorAcceleration,landscape,oldX,oldY);
moveTo(0,0,1,0,true, 10000, 10000); // Move mouse to position 0|0
delay(1000);
Serial.println("Moving mouse to the bottom right corner");
newX = screenWidth;
newY = screenHeight;
oldX = 0;
oldY = 0;
moveTo(newX,newY,cursorSpeed,cursorAcceleration,landscape,oldX,oldY);
delay(1000);
// Example 2 (doesn't require screen resolution and assumes a mouse speed of 1x):
//simpleMoveTo(500,500); // Move cursor to position 500|500
//delay(1000);
//simpleMoveTo(1000,10); // Move cursor to position 1000|10
//delay(1000);
simpleMoveTo(round(100/1.5),round(100/1.5));
delay(1000);
simpleMoveTo(round(522/1.5),round(421/1.5));
delay(1000);
simpleMoveTo(round(431/1.5),round(150/1.5));
delay(1000);
}
}
i saw about resetPosition on this ,https://github.com/T-vK/ESP32-BLE-Mouse/issues/5#issuecomment-672456215, and i am intersted in how implement it?i try some many way to abs mouse to iphone,but if i can reset position about mouse maybe i can use relative mode to mouse it with a few steps
@T-vK @ShiverZm : talking about BLE Touchscreen, I have tested this code-snippet https://github.com/NicoHood/HID/issues/123#issuecomment-723256512 and it looks great on (Android + Arduino Micro). The HID Descriptor (digitizer) and the HID.SendReport seems to work quite well and it could be great to implement it with ESP32-BLE-Mouse. Is there a simple way to test the Descriptor and the SendReport here ? Cheers.
emmm,you can modify the relative mouse descriptor from the mouse to pointer,you may be suprised.
@ShiverZm : did you make abs mouse working on ios with changing mouse to pointer ? I believe there is a way to have abs mouse working on ble for ios14
yes,i did it,but it worked not well,very slowly and cannot process much mouse data .i knew there must be a way,because our competitors they did it and worked very well.
@ShiverZm : thanks for the feedback, to confirm you had the absolute mode working ? Could you share your HID to make some tests of performance w/ ios14 ? several consumers companies got the abs ble mouse to work for ios14 so there is a way and it could great to find it w/ this lib
@xcarcelle https://www.codeproject.com/Articles/1001891/A-USB-HID-Keyboard-Mouse-Touchscreen-emulator-with
@xcarcelle my solutio is in it,other solution i never make it,if you successed,you could mail me for mail:[email protected]
@T-vK I test your moveTo and simpleMoveTo in the sequence, and delay(3000) in between steps:
1、 simpleMoveTo(0,0)
2、moveTo(100,100, 1.0, 0.0, 0, 0)
3、moveTo(200, 100, 1.0, 0.0, 100, 100)
4、moveTo(300, 100, 1.0, 0.0, 200, 100)
5、moveTo(400, 100, 1.0, 0.0, 300, 100)
6、moveTo(500, 100, 1.0, 0.0, 400, 100)
7、moveTo(600, 100, 1.0, 0.0, 500, 100)
8、moveTo(100, 100, 1.0, 0.0, 600, 100)
I noticed that after the 8th call. the mouse position will not go to (100, 100) as step 2 does. The test was done on ipad mini 5. iPados 15.6.1. I don't know how to fix such problem. Have any suggestions?
hi the program is great but what should i do to make the screen down with 1mm resolution
