tiny-pilot
Underscan causes skew in mouse position
Description
When the target computer displays an image with underscan (image size is smaller than the monitor size), then TinyPilot's mouse position gets distorted.
Toward the center of the screen, the mouse position is correct, but toward the outer edges, the mouse position is offset by a few dozen pixels.
What's the behavior that you expect?
TinyPilot forwards mouse position accurately even when the target display is in undesrcan.
What's happening instead?
Underscan causes a skew in the mouse position.
What are the steps to reproduce this behavior?
- Connect to a display that doesn't use the full screen resolution (e.g. BIOS for an ASRock AB350M-HDV motherboard)
- Move the mouse to the center of the screen
- Move the mouse to the edge of the screen
Screenshots
https://user-images.githubusercontent.com/7783288/112905596-aa459080-90b8-11eb-8fa5-5468ccbfdabf.mp4
Do you have an idea how to approach this? I see a similar thing with my security camera system, except that the point where the two pointers agree is not at the center, but instead it's near (but not at?!) the bottom right corner. The error at the top left corner is about 20% in both X and Y.
After a bit of lazy searching, I didn't find a place to hack at this - seems like mouse events just get scaled to [0..32767] and passed through to the gadget API without further processing. or did I miss it? and anyhow, if that scaling doesn't reach the corners of the screen, what else can you do? (obviously I don't understand HID)
The cleanest fix I can think of would be to crop the image in the browser to correct the underscan. We could maybe do it automatically by detecting the black border, but it's probably easier to have the user select the borders (using something like cropper.js).
That should correct the skew because then what the browser considers to be the 0,0 position would match the remote machine's true 0,0 position.
We'd want to make sure the solution also works for WebRTC video, as we're likely going to switch from the existing MJPEG to WebRTC.
