Support using android device's gyros as mouse, for aiming assist

[Williangalvani]

This allows for a Breath-of-the-wild-ish experience. This does confuse a couple of games such as Red dead redemption (where the tips keep switching between keyboard/mouse and joystick very fast).

This also makes using the touchscreen as mouse a bit weird.

Maybe this should add the mouse movement to the joystick readings instead?

anyway, opening this for visibility and feedback =]

Related to #1155 ?

[comanel]

@Williangalvani, i've noticed that when using Steam controller gyro set as mouse input, if i also send a low value analog input (like left trigger, but lower than any game's input threshold) at the same time, the interface won't switch from Xinput to KB/M. Could you pass on some ghost analog input value when you do the conn.SendMouseMove(x, y) ?

not to mention that an in-stream toggle would also be helpful (maybe mirror from turning mouse emulation on/off by holing the start button, one can do the same by holding the select button)

[Williangalvani]

@comanel great ideas, I'll try it. I also experimented with sending only joystick values instead, but that didn't quite do it either. I'll try your suggestions!

[csolisr]

Any plans to continue on this PR? Can we merge it as it currently is, and leave the "gyro as stick" to another PR?

[comanel]

Hi dude, GyroMouse thing works as it should, if you ignore the weird UI behavior switching from controller to keyboard prompts, it's good enough. Gyro as stick has some quirks though. Worst one i can think of is upon re-connecting to a running session, any continuous Joystick input isn't sent properly... it's like it's being interrupted each frame / packet.

Try merging them both, dive into it and see if there's anything you can improve.

On Wed, Jul 12, 2023 at 5:36 PM Carlos Solís @.***> wrote:

Any plans to continue on this PR? Can we merge it as it currently is, and leave the "gyro as stick" to another PR?

— Reply to this email directly, view it on GitHub https://github.com/moonlight-stream/moonlight-android/pull/1200#issuecomment-1632647496, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCVYQLBQITBUP75P4LW73TXP2Y6ZANCNFSM6AAAAAAXAJFYR4 . You are receiving this because you were mentioned.Message ID: @.***>

[cgutman]

It's a good PoC, but this PR can't be merged as is. There are a few issues:

• It changes the manifest so Moonlight won't install on devices without a gyroscope
• There's no way to actually turn it off (SensorEventListener is always registered using additional power)
• Sensor readings come in a fixed orientation and need to be adjusted for the orientation of the screen (landscape vs reverse landscape)
• It doesn't disable sensors when the activity enters PiP mode

Now that we can send motion events to the host natively using the new Sunshine protocol extensions, I think the best approach is to implement 4 gyro control options:

• Use gamepad motion sensors (already implemented)
• Use device motion sensors for gamepads that lack motion sensors
• Use device motion sensors for on-screen controls
• Use device motion sensors for mouse control

I suspect the first 3 will cover the vast majority of use cases. Since we're passing the gyroscope and accelerometer data to the host via the emulated DualShock 4 controller now in Sunshine nightly, users can do whatever stuff they want with it using Steam Input or similar tools. They can use it natively in games that support motion controls or map it to whatever control they want with Steam Input.

I can take a look at implementing these since I've been working on sensor stuff the last few days already.

[Williangalvani]

I'm sorry, I've been low on time and last time I tried, the docker I used to build the apk was no longer working properly, my apks were not getting sensor data, idk why. I'll try to pick this up again soon, but dealing with build issues is very frustrating.

[comanel]

I've tried the latest nightly for sunshine and built latest master on moonlight android. There is some sensor input read in Steam UI, as -0.1 acceleration on tilt axis inside the DS4 gyro calibration menu. allowUseOfGamepadMotionSensors is true, but something tells me that this has nothing to do with any sensors on the android device itself (duuh). Didn't get the chance to actually connect a controller with gyro sensors to the android device and see if the data passes through normally.

Any chance you could mirror device sensor data as you do with controller gyro sensor data ?

On Fri, Jul 14, 2023 at 8:04 AM Cameron Gutman @.***> wrote:

It's a good PoC, but this PR can't be merged as is. There are a few issues:

• It changes the manifest so Moonlight won't install on devices without a gyroscope
• There's no way to actually turn it off (SensorEventListener is always registered using additional power)
• Sensor readings come in a fixed orientation and need to be adjusted for the orientation of the screen (landscape vs reverse landscape)
• It doesn't disable sensors when the activity enters PiP mode

Now that we can send motion events to the host natively using the new Sunshine protocol extensions, I think the best approach is to implement 4 gyro control options:

• Use gamepad motion sensors (already implemented)
• Use device motion sensors for gamepads that lack motion sensors
• Use device motion sensors for on-screen controls
• Use device motion sensors for mouse control

I suspect the first 3 will cover the vast majority of use cases. Since we're passing the gyroscope and accelerometer data to the host via the emulated DualShock 4 controller now in Sunshine nightly, users can do whatever stuff they want with it using Steam Input or similar tools. They can use it natively in games that support motion controls or map it to whatever control they want with Steam Input.

— Reply to this email directly, view it on GitHub https://github.com/moonlight-stream/moonlight-android/pull/1200#issuecomment-1635276521, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCVYQJVMNX6OFIKI3HUFPLXQDHNJANCNFSM6AAAAAAXAJFYR4 . You are receiving this because you were mentioned.Message ID: @.***>

[cgutman]

Any chance you could mirror device sensor data as you do with controller gyro sensor data ?

Yep that's the plan.

[cgutman]

The latest code in master adds support for gamepad motion sensor emulation using device sensors (off by default). Let me know how that works for you all.

[Marocco2]

The latest code in master adds support for gamepad motion sensor emulation using device sensors (off by default). Let me know how that works for you all.

I've compiled and tried it on a Pixel 7 Pro but Steam won't recognize gyro even with DS4 and all options checked (Sunshine nightly)

[comanel]

Could you please share a link with the android build, build on appveyor and I can't build it myself too soon..

On Sun, Sep 17, 2023, 12:33 Marocco2 @.***> wrote:

The latest code in master adds support for gamepad motion sensor emulation using device sensors (off by default). Let me know how that works for you all.

I've compiled and tried it on a Pixel 7 Pro but Steam won't recognize gyro even with DS4 and all options checked (Sunshine nightly)

— Reply to this email directly, view it on GitHub https://github.com/moonlight-stream/moonlight-android/pull/1200#issuecomment-1722436129, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCVYQNIMVFNWWA44G7F7E3X227XHANCNFSM6AAAAAAXAJFYR4 . You are receiving this because you were mentioned.Message ID: @.***>

[Marocco2]

The latest code in master adds support for gamepad motion sensor emulation using device sensors (off by default). Let me know how that works for you all.

I've compiled and tried it on a Pixel 7 Pro but Steam won't recognize gyro even with DS4 and all options checked (Sunshine nightly)

Nevermind. Apparently it doesn't work with touch controls. Attaching a real controller and disabling gamepad motion but leaving emulation enabled will work as intended

[Marocco2]

Could you please share a link with the android build, build on appveyor and I can't build it myself too soon..

I can share my compiled version: https://send.vis.ee/download/24e7ddf2be463119/#QGyF4fUiOUAJ_9vh13L6Ug

[comanel]

Thanks, unfortunately this doesn't work for me either, using a oneplus 7t pro paired with a razer kishi. Ds4 gyro is passed through as it was in previous builds though.

On Sun, Sep 17, 2023, 13:27 Marocco2 @.***> wrote:

Could you please share a link with the android build, build on appveyor and I can't build it myself too soon.. … <#m_-2870162213784852305_> On Sun, Sep 17, 2023, 12:33 Marocco2 @.> wrote: The latest code in master adds support for gamepad motion sensor emulation using device sensors (off by default). Let me know how that works for you all. I've compiled and tried it on a Pixel 7 Pro but Steam won't recognize gyro even with DS4 and all options checked (Sunshine nightly) — Reply to this email directly, view it on GitHub <#1200 (comment) https://github.com/moonlight-stream/moonlight-android/pull/1200#issuecomment-1722436129>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCVYQNIMVFNWWA44G7F7E3X227XHANCNFSM6AAAAAAXAJFYR4 https://github.com/notifications/unsubscribe-auth/ACCVYQNIMVFNWWA44G7F7E3X227XHANCNFSM6AAAAAAXAJFYR4 . You are receiving this because you were mentioned.Message ID: @.>

I can share my compiled version: https://send.vis.ee/download/24e7ddf2be463119/#QGyF4fUiOUAJ_9vh13L6Ug

— Reply to this email directly, view it on GitHub https://github.com/moonlight-stream/moonlight-android/pull/1200#issuecomment-1722445764, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCVYQJNPH337T2B6ZLTRG3X23GB3ANCNFSM6AAAAAAXAJFYR4 . You are receiving this because you were mentioned.Message ID: @.***>

[comanel]

Boy do I feel stupid, forgot to check the frigging box :)) Works wonderfully!

On Sun, Sep 17, 2023, 13:43 Adrian Coman @.***> wrote:

Thanks, unfortunately this doesn't work for me either, using a oneplus 7t pro paired with a razer kishi. Ds4 gyro is passed through as it was in previous builds though.

On Sun, Sep 17, 2023, 13:27 Marocco2 @.***> wrote:

Could you please share a link with the android build, build on appveyor and I can't build it myself too soon.. … <#m_-4829164415291168043_m_-2870162213784852305_> On Sun, Sep 17, 2023, 12:33 Marocco2 @.> wrote: The latest code in master adds support for gamepad motion sensor emulation using device sensors (off by default). Let me know how that works for you all. I've compiled and tried it on a Pixel 7 Pro but Steam won't recognize gyro even with DS4 and all options checked (Sunshine nightly) — Reply to this email directly, view it on GitHub <#1200 (comment) https://github.com/moonlight-stream/moonlight-android/pull/1200#issuecomment-1722436129>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCVYQNIMVFNWWA44G7F7E3X227XHANCNFSM6AAAAAAXAJFYR4 https://github.com/notifications/unsubscribe-auth/ACCVYQNIMVFNWWA44G7F7E3X227XHANCNFSM6AAAAAAXAJFYR4 . You are receiving this because you were mentioned.Message ID: @.>

I can share my compiled version: https://send.vis.ee/download/24e7ddf2be463119/#QGyF4fUiOUAJ_9vh13L6Ug

— Reply to this email directly, view it on GitHub https://github.com/moonlight-stream/moonlight-android/pull/1200#issuecomment-1722445764, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCVYQJNPH337T2B6ZLTRG3X23GB3ANCNFSM6AAAAAAXAJFYR4 . You are receiving this because you were mentioned.Message ID: @.***>