Use the HTML5 Gamepad API instead of uinput and udev for controllers

[ehfd]

Currently, selkies-gstreamer does support gamepads only when the uinput device is provisioned. This is very cumbersome and risky in Kubernetes.

Instead of using that, inputs can be translated within the web page using JavaScript and be sent to the host instead for pynput to translate them to Xorg operations (and later Wayland #46 using virtual keyboard), with the Gamepad API of web browsers.

https://developer.mozilla.org/en-US/docs/Web/API/Gamepad_API/Using_the_Gamepad_API https://gamepad-tester.com

Relevant files: gst-web module and webrtc_input.py.

[ehfd]

Requires a careful assessment of the coordination with a Gamepad input and transforming them into keyboard or mouse signals or using xserver-xorg-input-joystick without using modprobe or touching the kernel.

[ehfd]

It's not impossible to just translate the Gamepad API signal within the browser and then pass like it was a keystroke, as long as careful tuning is done.

A couple of programs exist to map gamepad buttons to keyboard keys, including:

qjoypadAUR antimicroxAUR sc-controllerAUR steam - see Steam#Steam Input All work well without the need for additional X.org configuration.

https://wiki.archlinux.org/title/Gamepad

[loochek]

If you are interested, I can suggest an another approach for games running under Wine. I've written a small Xinput reimplementation that acts like a virtual Xbox gamepad and can be controlled from host Linux environment via a named pipe. So, I've successfully paired it with your project (I can provide dirty rewritten version of webrtc_input.py). As games are often run under Wine, my solution is applicable quite widely and is much better than translating gamepad input to the keyboard and mouse.

[ehfd]

@loochek Hi! Thanks for bringing this up for us! We are short of hands, so this is a bit behind priority for us now. If you can provide us your webrtc_input.py file, someone might be able to clean it so that it doesn't collide with existing flows and make a PR for us. Thanks a lot for implementing this solution.

[loochek]

@ehfd webrtc_input.py Basically I just replaced the logic in __js_connect and __js_emit with my own

[ehfd]

Thanks! @loochek

[JinhuaSu]

@ehfd webrtc_input.py Basically I just replaced the logic in __js_connect and __js_emit with my own

Thanks! It works on me. Binding of Issac: XBOX mode Genshin Impact: USB controller mode

[ehfd]

Update: Perhaps, try using https://github.com/ValveSoftware/gamescope as an alternative.

https://github.com/akdor1154/gamescope-pkg

[ehfd]

Relevant: https://web.archive.org/web/20211003152946/https://parsec.app/blog/game-streaming-tech-in-the-browser-with-parsec-5b70d0f359bc

[danisla]

Selkies already uses the HTML5 Gamepad API, so there is nothing to change.

I am not aware of any way to support gamepads without uinput, Joystick events are not handled by Xorg.

[ehfd]

1. Perhaps, find some kind of a way to create a synthetic /dev/input/* device that passes input from HTML5 without /dev/uinput (or an emulated /dev/uinput). I agree this is hard.

Alternative to evdev: perhaps joydev?

2. If not number 1, you are right. The only other option to joystick events with uinput is to replace them with cursor/key events. Since we already are obtaining input with the HTML5 Gamepad API, it would be easier to just translate them to key events.

joystick is an Xorg input driver for controlling the pointer with a joystick device.

Use this driver, if you want to

• generate cursor movement, perform button or key events to control desktop and applications
• generate cursor key events for playing legacy games, that have no native joystick support

Do not use, if you want to

• play games, that have native joystick support
• use XI2 applications. The evdev(4) driver will suffice for those in most cases.

3. Even when the above options are exhausted, it would be possible to improve things at least within Wine. There are applications that simply do not input correctly even using a mouse in Wine.

https://github.com/loochek/wine-vpad/ https://gist.github.com/loochek/f0d875fb8c2adcb92431e13868cbd079

Either way, I'm still keeping this issue open.

The documentation to use uinput in bare metal is still required. #90 #28

[danisla]

I’m looking into ways to emulate the js0 evdev with an LD_PRELOAD shim. Something that will intercept the syscalls and connect to the Selkies gstreamer app.

[ehfd]

I’m looking into ways to emulate the js0 evdev with an LD_PRELOAD shim. Something that will intercept the syscalls and connect to the Selkies gstreamer app.

Something similar to VirtualGL's internal mechanics, I guess.

[ehfd]

We have a completely novel implementation of gamepad support in containers with #95. Please leave a message here or in the Discord if you would like to help test Wine applications.