Support game controllers

[LHLaurini]

closes #99

What it does

• Handles events from connected game controllers;
• Transfers game controller events to Android device;
• Emulates an Xbox 360 game controller input device.

Important stuff

• In order to emulate input devices, we need a few system calls (open, ioctl, write and close). Since we can't use most of these directly from Java (AFAIK), we need either native code or a library that allows us to call these functions. To avoid depending on the NDK, I'm using the Java Native Access library;
• JNA requires a native library (libjnidispatch.so). For some reason, app_process doesn't want to load it, so right now it's extracted to /data/local/tmp on initialization and cleaned up later.

Things you can help with

• Testing:
  • Comment whether it works on your device or not, so I can keep a list;
  • Axes behavior;
  • Multiple controllers.
• Find a way to use the ioctl syscall without native code;
• Load native libraries without extracting them;
• Figure out multiple controllers with event injection.

To-do

• [x] Add command line option for disabling game controllers
• [x] Write tests
• [x] Isolate uinput related stuff into another class (so other types of devices can be implemented)
• [x] Fix input not working when uinput fails
• [x] Support legacy uinput
• [x] Look into event injection (for when uinput is unavailable)
• [ ] Look into getting games to recognize axis movement when using event injection
• [ ] Implement controller through event injection

Out of scope

I probably won't be adding some special features to this pull request, such as:

• Vibration
• Battery reporting
• Lights
• Touchpad

Feedback

If you have any feedback (suggestions, complaints, new info, ...), please don't hesitate to write it down below.

Tested devices

Manufacturer	Model	Android version	Last tested on	Tested by	Supported	Supported with root
Motorola	Moto G5 Plus	8.1	2021-04-30	@LHLaurini	:x:	:grey_question:
Motorola	moto g(6) play	9	2021-04-30	@LHLaurini	:x:	:grey_question:
Samsung	Galaxy A7 2018	10	2021-04-06	@Fablo020	:x:	:grey_question:
Samsung	Galaxy M21s	11	2021-04-30	@LHLaurini	:heavy_check_mark:	:grey_question:
Huawei	Honor Play COR-L29	9	2021-05-01	@Cannahawk	:x:	:grey_question:
Xiaomi	Pocophone F1	11	2021-05-23	@Nightm4res	:grey_question:	:heavy_check_mark:
Sony	Xperia XZ1 Compact	11	2021-06-14	@Schlumpf7	:heavy_check_mark:	:grey_question:
Sony	Xperia XZ	:grey_question:	2021-06-15	@Schlumpf7	:heavy_check_mark:	:grey_question:
Blackview	BV6000	:grey_question:	2021-06-15	@Schlumpf7	:heavy_check_mark:	:grey_question:
Xiaomi	Mi A3	11	2021-06-17	@RuiGuilherme	:heavy_check_mark:	:grey_question:
Nvidia	Shield TV	11	2022-02-12	@scaldav	:heavy_check_mark:	:grey_question:
Blackview	BV6300 Pro	10	2022-05-02	@Fancy2209	:heavy_check_mark:	:grey_question:
Google	Pixel 6	12	2022-06-02	@AntoninHuaut	:heavy_check_mark:	:grey_question:
Redmi	Note 9 Pro 5G	12	2022-06-18	@esec	:heavy_check_mark:	:grey_question:
Redmi	Note 5 Pro	11*	2022-06-20	@Sigmacringe	:heavy_check_mark:	:grey_question:

The list of (un)supported devices can change anytime.

Reporting a new device

If you have tested with a device not on the list, please report it by writing a comment below. Please follow (more or less) the following format:

**Manufacturer**: Foo
**Model**: Bar 12
**Android version**: 123.4 (specify if not stock)
**Root**: Y/N

NEW: Windows and Linux releases

Prebuilt versions of scrcpy with the new patches can be found here. Remember to set the SCRCPY_SERVER_PATH variable.

[LHLaurini]

Injecting KeyEvent (already used for keyboard) should also allow to inject gamepad buttons

For some reason, that didn't even cross my mind. I'll look into it.

[LHLaurini]

Ok, so after a little research, here what I've found so far:

• Using IInputManager.injectInputEvent to inject KeyEvents and MotionEvents wouldn't work with multiple controllers, unless we can somehow register new InputDevices;
• InputDriver looked promising until I realized it only works with Android Things.

I'll keep looking when I have some free time.

[rom1v]

https://github.com/Genymobile/scrcpy/blob/dce08677375dbb1b65217c6f4080b012284f4afa/server/src/main/java/com/genymobile/scrcpy/Controller.java#L210-L214 https://github.com/Genymobile/scrcpy/blob/dce08677375dbb1b65217c6f4080b012284f4afa/server/src/main/java/com/genymobile/scrcpy/Device.java#L178-L179

InputDevice.SOURCE_GAMEPAD and pass some arbitrary deviceId?

[LHLaurini]

InputDevice.SOURCE_GAMEPAD

Yes, we'd need to use InputDevice.SOURCE_GAMEPAD, InputDevice.SOURCE_JOYSTICK and InputDevice.SOURCE_DPAD, depending on the kind of event.

and pass some arbitrary deviceId?

That was my first idea, but it still won't allow apps to query info about controllers. I'm going to implement and test it, maybe it works better than I think.

[Looneybinke903]

But in this case it is only working through code or already has an executable to download?

because if it is a code you could tell me where to put it and how !?

[LHLaurini]

But in this case it is only working through code or already has an executable to download? because if it is a code you could tell me where to put it and how !? …

No, there's no binary release yet. You can try compiling my branch. Take a look at the build instructions, they're pretty easy to follow. Just don't use the prebuilt server.

Note: all of this is still work in progress, so don't expect it to be perfect.

[LHLaurini]

I implemented event injection, as suggested, but I can't seem to find a way to set axes from MotionEvent.obtain and there's no setAxis method either, so I just tested button events for now.

• Using the Gamepad Tester app, I can see the correct events for buttons, but, of course, it reports "Name: Virtual";
• gamepad-tester.com on Google Chrome doesn't work anymore. Just says "No gamepad detected.". That's because the HTML5 Gamepad API needs to detect controllers for them to work;
• I've tried the game "Into the Dead", buttons work;
• I'll try other games. I don't expect this approach to work with Unity games due to the way it handles controllers.

Even if it works, I still need to figure out MotionEvent.

Now, to be honest, I still think the uinput approach is the best we got, as long as we find a way to get app_process to properly load native libraries.

[LHLaurini]

Tested both methods with "Dead Trigger" (an Unity game). With uinput, controls work out of the box. By injecting events, every single button needs to be rebound (I'm surprised it works at all).

[rom1v]

Hmm, opening /dev/uinput requires root permissions, right?

[LHLaurini]

Hmm, opening /dev/uinput requires root permissions, right?

Nope. Just limited to the ADB shell (at least on my phone).

Here, look:

f41:/dev $ ls -l /dev/uinput                                                                                           
crw-rw---- 1 uhid uhid 10, 223 2021-02-12 13:43 /dev/uinput
f41:/dev $ whoami
shell
f41:/dev $ groups
shell input log adb sdcard_rw sdcard_r ext_data_rw ext_obb_rw net_bt_admin net_bt inet net_bw_stats readproc uhid

shell is a member of the uhid group.

[rom1v]

Hmm, opening /dev/uinput requires root permissions, right?

Nope. Just limited to the ADB shell (at least on my phone).

Does it mean it can also be used to inject key/touch events and avoid (in some way) the ASCII-limitation of Android text injection?

https://github.com/Genymobile/scrcpy/blob/master/FAQ.md#special-characters-do-not-work

[LHLaurini]

Does it mean it can also be used to inject key/touch events and avoid (in some way) the ASCII-limitation of Android text injection?

Yes, it can. When I started writing my patch, I tried injecting key presses and it worked.

Also, since Android would recognize the virtual device as if it were real, it should also allow the user to select a text box and not have the virtual keyboard pop up. In fact, I'm pretty sure I'd seen the "Configure physical keyboard" notification.

[LHLaurini]

I decided to make some changes so that a future pull request can add support for keyboard (or even mouse) using uinput.

[rom1v]

You might be interested in android.system.Os. The documentation does not show all the methods, because some are hidden, but you can probably still access them. For example ioctlInt().

[rom1v]

Or even Libcore.java + Linux.java.

With that, you might not need JNA.

EDIT: to be adapted on older versions of Android, for example on Android 6 there is no field rawOs, but on Android 10 I have access to it and all the methods.

[LHLaurini]

You might be interested in android.system.Os. The documentation does not show all the methods, because some are hidden, but you can probably still access them. For example ioctlInt().

Thanks. Will look into it.

[LHLaurini]

You might be interested in android.system.Os. The documentation does not show all the methods, because some are hidden, but you can probably still access them. For example ioctlInt().

Or even Libcore.java + Linux.java.

With that, you might not need JNA.

EDIT: to be adapted on older versions of Android, for example on Android 6 there is no field rawOs, but on Android 10 I have access to it and all the methods.

Good find.

~We could use fcntlInt for UI_SET_EVBIT, UI_SET_KEYBIT and UI_SET_ABSBIT, and use fcntlVoid for UI_DEV_CREATE and UI_DEV_DESTROY.~

We could use ioctlInt for UI_SET_EVBIT, UI_SET_KEYBIT and UI_SET_ABSBIT, and it may also work for UI_DEV_CREATE and UI_DEV_DESTROY (I had mixed up ioctl and fcntl).

Sadly, UI_DEV_SETUP and UI_ABS_SETUP require pointers, so these methods won't work. If there were a version that received a byte[] we could use it.

Not all hope is lost, however. We should be able to use an older setup method, in which we write a struct uinput_user_dev. Hopefully it still works.

Another problem is that we'll need to be careful with the structs, since we'll have to assemble them manually. At least, if we do something wrong, it shouldn't crash, just won't work.

Finally, this is going to take me a while to implement, but at least we won't need JNA.

[LHLaurini]

Hmm, I can't seem figure out how to use ioctlInt.

import android.system.Os;
Os.ioctlInt(fd, UI_SET_EVBIT, EV_KEY);

fails with

error: cannot find symbol
        Os.ioctlInt(fd, UI_SET_EVBIT, EV_KEY);
          ^
  symbol:   method ioctlInt(FileDescriptor,int,int)
  location: class Os

Method ioctlInt = Os.class.getDeclaredMethod("ioctlInt", int.class/*, int.class*/);

throws

java.lang.NoSuchMethodException: android.system.Os.ioctlInt [int]
        at java.lang.Class.getMethod(Class.java:2072)
        at java.lang.Class.getDeclaredMethod(Class.java:2050)
        at com.genymobile.scrcpy.GameController.getIoctlInt(GameController.java:134)
        at com.genymobile.scrcpy.GameController.<clinit>(GameController.java:143)
        at com.genymobile.scrcpy.Controller.handleEvent(Controller.java:177)
        at com.genymobile.scrcpy.Controller.control(Controller.java:75)
        at com.genymobile.scrcpy.Server$2.run(Server.java:130)
        at java.lang.Thread.run(Thread.java:923)

import libcore.io.Libcore;

fails with

error: package libcore.io does not exist
import libcore.io.Libcore;
                 ^

open, write and close seem to be working just fine. Any ideas?

[rom1v]

(Quick msg just to answer about your blocking points)

Before this commit the signature was:

public int ioctlInt(FileDescriptor fd, int cmd, Int32Ref arg) …

So:

Method m = Os.class.getDeclaredMethod("ioctlInt", FileDescriptor.class, int.class, Class.forName("android.system.Int32Ref"));

For LibCore/Linux:

Object linux = Class.forName("libcore.io.Libcore").getDeclaredField("rawOs").get(null);
Class<?> linuxClass = linux.getClass();

[LHLaurini]

(Quick msg just to answer about your blocking points)

Before this commit the signature was:

public int ioctlInt(FileDescriptor fd, int cmd, Int32Ref arg) …

So:

Method m = Os.class.getDeclaredMethod("ioctlInt", FileDescriptor.class, int.class, Class.forName("android.system.Int32Ref"));

For LibCore/Linux:

Object linux = Class.forName("libcore.io.Libcore").getDeclaredField("rawOs").get(null);
Class<?> linuxClass = linux.getClass();

Ah, I see, I completely forgot the first parameter and was using the wrong one for the third. Thank you for the help.

So now I'm able to call ioctlInt, but there's a problem. Int32Ref is a pointer to an integer, because ioctlInt uses the third parameter to "return" an integer. That's why they've now changed the signature. Even if I try to initialize it with the correct value, I just get ioctl failed: EINVAL (Invalid argument).

I only found ioctlInt used with SIOCINQ, SIOCOUTQ and FIONREAD which are all "getters". ioctlInetAddress is also used with getters, only they use struct ifreq internally.

I'll keep looking.

PS: The pointer is internal, so we can't try to manipulate it somehow. Also, I found nothing new by searching for "ioctl java". Native may really be the only way.

[quyleanh]

@LHLaurini are you still working on this PR?

[LHLaurini]

@LHLaurini are you still working on this PR?

Not right now, but it works well in its current state, AFAIK. What I have left to do is:

• make some changes so this can also be used for proper key injection (maybe even mouse actions);
• investigate some more about a way to call ioctl without JNA (or JNI), which may not be possible;
• add an option for disabling controller input.

I'll work some more on this when I have some free time, maybe tomorrow. If you want to use this branch, you'll have to compile it yourself. Feel free to ask any questions.

[quyleanh]

@LHLaurini thank you for your response. Actually, I would like to use this branch with UTF-8 text injection. So I think I will wait until you implement the first item.

[LHLaurini]

@LHLaurini thank you for your response. Actually, I would like to use this branch with UTF-8 text injection. So I think I will wait until you implement the first item.

I don't think my changes will allow for UTF-8 text injection (at least not directly), if you need to have a function that injects a string.

But if you mean being able to use any keyboard (with accents and other special keys) and have it be recognized by the system as a physical device, then sure.

I'm not sure which one you meant.

[quyleanh]

I'm not sure which one you meant.

My purpose is whenever I type the UTF-8 characters from physical keyboard, the system will receive it and show correctly on device. May I ask your being able to use any keyboard (with accents and other special keys) and have it be recognized by the system as a physical device function is implemented or not yet?

[LHLaurini]

My purpose is whenever I type the UTF-8 characters from physical keyboard, the system will receive it and show correctly on device.

Ah, I see. In that case, character encoding (like UTF-8) has nothing to do with the problem. When that's implemented, scrcpy would just forward keystrokes as-is to the device as if the keyboard is connected to it.

May I ask your being able to use any keyboard (with accents and other special keys) and have it be recognized by the system as a physical device function is implemented or not yet?

Not yet and I don't intend to implement it in this pull request.

The best method for injecting input in any Linux based system is uinput. Currently, this branch uses it to send just controller input. I intend to isolate uinput stuff from the GameController class (soon) and make it so it can eventually also be used for keyboard injection (in a future PR).

[quyleanh]

@LHLaurini thank you for your response. I'm looking forward to hearing news from your PR.

[quyleanh]

@LHLaurini Thank you for your update. I see you have separated the uinput to UinputDevice. I would like to test the keyboard injection things we discussed above, so please let me know when your branch is ready and can buildable.

[LHLaurini]

I have now implemented all that I wanted to. Since it seems we won't be able to use evdev with pure Java, all that's left is to decide about the native interface. We have two options:

• Use JNA, adding a dependency. I could add a compile-time switch (--with[out]-jna), making it optional;
• Use JNI and write a simple C (or C++) wrapper for the needed low-level functionality. In this case we'd depend on the NDK (which is quite heavy). We could also prebuild these libraries, same as the scrcpy-server.

What do you think, @rom1v ?

[Fablo020]

Hello, thanks for releasing this. I just bought an Linux Server to build this. After getting many Issues, I finally managed to build this.

I tried it on my Phone but it does not work. Not on my Homescreen, Apps with Controller Support and "Gamepad testing" apps I use an original Xbox 360 Controller with an original USB Dongle from Microsoft. Phone: A7 2018, Andoid 10 connected via USB or using adb connect. Latest build from your fork (release-v1.17-6-gf8524a2) on Windows 10