moonlight-stream
[Issue]: pen/touch lag sometimes
Describe the bug
The stylus/touch function of android pad connected to Windows/Linux has some bugs. Other functions are good, and the stylus/touch works well most of the time, but it will freeze every ten to twenty seconds (while other content will not freeze, only the pen/touch's). When the pen/touch freezes, the movement of the pen/touch (that is, writing and the pen cursor and touch) will be intermittent. It will return to normal after two or three seconds.
Steps to reproduce
Pad: Xiaomi android pad Computer: Windows 11 pro / Ubuntu22.04, ASUS Theone 2 laptop Windows / Linux starts Sunshine (NOT Nvidia Shield) and android pad connects to it using WiFi in an LAN. Use virtual display and map it to android pad as if the pad is another display of Windows with pen and touch features. Or simply connect to an existing screen. Use a pen to write any content to any writable apps, such as Onenote and Photoshop (even just use the pen to contact the screen and move it continuously). Or just draw using finger.
Affected games
Desktop. Including any using of pen/touch, such as onenote and Photoshop.
Other Moonlight clients
No
Moonlight adjusted settings
No
Moonlight adjusted settings (please complete the following information)
Resolution: 3048x2032 Refresh rate: 144Hz Bitrate: 96
Moonlight default settings
Yes
Gamepad-related connection issue
No
Gamepad-related input issue
Yes
Gamepad-related streaming issue
No
Android version
Android 14
Device model
Xiaomi Pad 6S Pro
Server PC OS version
Windows 11 23H2 / Ubuntu22.04
Server PC GeForce Experience version
10.0.2.210
Server PC Nvidia GPU driver version
565.90
Server PC antivirus and firewall software
Huorong
Screenshots
No response
Relevant log output
No response
Additional context
No response
https://github.com/user-attachments/assets/d63373ed-ada6-4a14-a97a-c73fff69a163
I ran another test streaming over the Internet instead of the LAN.
The results are weird. Although the frame rate has dropped, the network is not stable, and there is a problem of insufficient bit rate, and the stylus freezes more frequently and for longer periods of time, the written pattern and words do not have problems like in the LAN, and the handwriting is actually fine.
I have the same problems as the original author. ~~As far as I have found out during testing, it is because the ControlEvents are sent via UDP.~~ ~~As soon as they are sent via TCP, the problem no longer occurs. At least that's how it seems in my tests.~~ ~~(I used an OpenVPN server that works via TCP to tunnel the UDP packets via TCP from Android Moonlight to Sunshine on Windows)~~
~~Is it possible to modify Moonlight/Sunshine so that the user can choose whether the ControlEvents are sent via TCP/UDP?~~
The problem is that move events are batched for pens and move events are not sent reliably. Both issues are fixed by the above pull requests.