Support game controllers #2130
Support game controllers #2130
Conversation
There was a problem hiding this comment.
Thank you for your contribution.
In order to emulate input devices, we need a few system calls (
open,ioctl,writeandclose). Since we can't use most of these directly from Java (AFAIK), we need either native code or a library that allows us to call these functions.
Injecting KeyEvent (already used for keyboard) should also allow to inject gamepad buttons, don't they?
For example: KEYCODE_BUTTON_L1, KEYCODE_BUTTON_START, KEYCODE_BUTTON_X…
For some reason, that didn't even cross my mind. I'll look into it. |
|
Ok, so after a little research, here what I've found so far:
I'll keep looking when I have some free time. |
|
scrcpy/server/src/main/java/com/genymobile/scrcpy/Controller.java Lines 210 to 214 in dce0867 scrcpy/server/src/main/java/com/genymobile/scrcpy/Device.java Lines 178 to 179 in dce0867
|
|
Yes, we'd need to use
That was my first idea, but it still won't allow apps to query info about controllers. I'm going to implement and test it, maybe it works better than I think. |
|
But in this case it is only working through code or already has an
executable to download?
because if it is a code you could tell me where to put it and how !?
… |
No, there's no binary release yet. You can try compiling my branch. Take a look at the build instructions, they're pretty easy to follow. Just don't use the prebuilt server. Note: all of this is still work in progress, so don't expect it to be perfect. |
|
I implemented event injection, as suggested, but I can't seem to find a way to set axes from
Even if it works, I still need to figure out Now, to be honest, I still think the uinput approach is the best we got, as long as we find a way to get |
|
Tested both methods with "Dead Trigger" (an Unity game). With uinput, controls work out of the box. By injecting events, every single button needs to be rebound (I'm surprised it works at all). |
|
Hmm, opening /dev/uinput requires root permissions, right? |
Nope. Just limited to the ADB shell (at least on my phone). Here, look:
|
Does it mean it can also be used to inject key/touch events and avoid (in some way) the ASCII-limitation of Android text injection? https://github.com/Genymobile/scrcpy/blob/master/FAQ.md#special-characters-do-not-work |
Yes, it can. When I started writing my patch, I tried injecting key presses and it worked. Also, since Android would recognize the virtual device as if it were real, it should also allow the user to select a text box and not have the virtual keyboard pop up. In fact, I'm pretty sure I'd seen the "Configure physical keyboard" notification. |
|
I decided to make some changes so that a future pull request can add support for keyboard (or even mouse) using uinput. |
|
You might be interested in |
|
Or even Libcore.java + Linux.java. With that, you might not need JNA. EDIT: to be adapted on older versions of Android, for example on Android 6 there is no field |
Thanks. Will look into it. |
Good find.
We could use Sadly, Not all hope is lost, however. We should be able to use an older setup method, in which we Another problem is that we'll need to be careful with the structs, since we'll have to assemble them manually. At least, if we do something wrong, it shouldn't crash, just won't work. Finally, this is going to take me a while to implement, but at least we won't need JNA. |
|
Hmm, I can't seem figure out how to use import android.system.Os;
Os.ioctlInt(fd, UI_SET_EVBIT, EV_KEY);fails with Method ioctlInt = Os.class.getDeclaredMethod("ioctlInt", int.class/*, int.class*/);throws import libcore.io.Libcore;fails with
|
|
(Quick msg just to answer about your blocking points) Before this commit the signature was: public int ioctlInt(FileDescriptor fd, int cmd, Int32Ref arg) …So: Method m = Os.class.getDeclaredMethod("ioctlInt", FileDescriptor.class, int.class, Class.forName("android.system.Int32Ref"));For LibCore/Linux: Object linux = Class.forName("libcore.io.Libcore").getDeclaredField("rawOs").get(null);
Class<?> linuxClass = linux.getClass(); |
Ah, I see, I completely forgot the first parameter and was using the wrong one for the third. Thank you for the help. So now I'm able to call I only found I'll keep looking. PS: The pointer is internal, so we can't try to manipulate it somehow. Also, I found nothing new by searching for "ioctl java". Native may really be the only way. |
|
@LHLaurini are you still working on this PR? |
Not right now, but it works well in its current state, AFAIK. What I have left to do is:
I'll work some more on this when I have some free time, maybe tomorrow. If you want to use this branch, you'll have to compile it yourself. Feel free to ask any questions. |
|
@LHLaurini thank you for your response. |
I don't think my changes will allow for UTF-8 text injection (at least not directly), if you need to have a function that injects a string. But if you mean being able to use any keyboard (with accents and other special keys) and have it be recognized by the system as a physical device, then sure. I'm not sure which one you meant. |
|
I rebased this onto the latest dev branch: https://github.com/Genymobile/scrcpy/compare/dev...yume-chan:scrcpy:feat/game-controller?expand=1 Next will try to create a virtual keyboard and using |
2023-02-07.17-32-59.mp4It works, slowly. After all, this feature has been deprecated since Android 3.
EDIT2: If I add two virtual keyboards, the second one will be classified as physical keyboard. This is because the first one is recognized as built-in keyboard, then got rejected due to its device id not being zero. Android is such an interesting OS. However, still need to disable Android IME for Unicode injection to work. |
|
Hello @LHLaurini! Is the current state of your repository fork works under scrcpy 1.19? I recently switch to Android 12 and was sad to see it could not run anymore with your release. I tried to setup and configurate a clean Linux environment to compile your fork, scrcpy is running again but controller inputs doesn't seem to be recognized, any quick ideas? If an easy fix is possible, might be possible I do an update to my old video explaining quickly how to setup your scrcpy gamepad-compatible (https://www.youtube.com/watch?v=zeJiB4IUBdc, credited in the description ofc) Have a good week-end, cheers! |
Hi, there. I stopped working on this, but the fork should still be functional (but out-of-date). You can try following the troubleshooting steps I mentioned in other comments or try out @yume-chan 's fork. |
I will try this method on my uhid demo now. It would be fantastic if it works! |
This is the way to set axis values for a MotionEvent. Using PointerProperties and PointerCoords. |
|
Hello! The "uinputcommand_jni" library is already built into Android 12 and higher versions of the Android system. We can directly load the "uinputcommand_jni" library and use its JNI functions to register and use uinput devices with pure Java code. However, if you need better compatibility with lower versions of Android systems, I suggest using the "hidcommand_jni" library, which has been built into the system since Android 6.0.1. You can find information on how to use the JNI functions in "uinputcommand_jni" library and "hidcommand_jni" library in the Android source code links below: Uinput and uhid can achieve almost the same effect, but there is a subtle difference, which is that uinput can ignore the logical maximum and minimum value limits of axis events. Therefore, I think adding both of these methods to the source code of scrcpy would be more comprehensive. In addition, I already have a simple demo that uses pure Java to register and use uhid devices by loading the "hidcommand_jni" library. You can check it out here: https://github.com/WuDi-ZhanShen/AndroidUHidPureJava Finally, in the initial version of my Android app, I also used uinput to implement the function. However, according to user feedback, on some devices, there was no "/dev/uinput" node file, so uinput did not work at all. After switching to uhid, there was no such problem. Because uhid is responsible for registering all Bluetooth gamepads, Bluetooth keyboards, and Bluetooth mice, all Android devices that support Bluetooth must also support uhid. |
I hope someone can pick up the work for scrcpy. |
|
Manufacturer: POCO (Xiaomi) |
|
Manufacturer: Google Everything works fine on all builds of 14 Beta. Playing call of duty warzone mobile with no problem using my xinput controller. Will update when android 14 is out. |
|
Manufacturer: Samsung Every button works perfectly on both wifi and usb, It stutters over 2.4GHz wifi but works perfectly on a direct connection over hotspot. just a "feature" i found: if you start this while focusing on another window, you can use your gamepad without having to be focused on the main scrcpy window. also, by creating: C:\usr\x86_64-w64-mingw32\share\scrcpy I would love to see this updated with the latest scrcpy version for newer features. |
|
Manufacturer: Oneplus Its working, I do have ds4windows installed. I dont know if it helps. |
|
Manufacturer: Samsung EDIT: Gamepad works fine with this build. I get no sound though. I have to spin up a newer instance of scrcpy in the background to get audio from my PC. Would be nice for them to merge the gamepad controls into the main release. |
Yeah, would love to see this upstreamed to the latest version. |
|
Now that UHID (keyboard and mouse) is available (#4473), can this work be rebased on the latest version? |
This implementation uses #4473, on the other hand, works by using So yes and no. Some of the code could be reused, but a decent portion would have to be rewritten. To put it bluntly, that's a non-trivial amount of work I currently have neither the time nor the interest for. That may change one day but, in the meantime, if someone is willing to put in the work, please feel free to use my code as a starting point. I'm closing this pull request for now. Thank you to everybody who provided feedback. |
|
So is this abandoned @LHLaurini ? I have a remote controller from parsec/sunshine connected on my pc whose inputs need to be passed to my mirrored android phone running the game |
The main difference is that it injects events in UHID format to The reason why it is problematic to use uinput is that it requires a call to What is missing to implement it in UHID is a report descriptor for a gamepad and the matching HID events format for simulating a gamepad on Android. Here is how it is done for keyboard and mouse for reference:
Adapting it to gamepads requires to read and understand some HID specs:
That's an implementation detail: the HID events are generated on the client side to share the code with AOA (to send HID events directly over USB). If you (or anyone else) can adapt your PR to inject UHID events from Java (basically your |
I read the specs, and wrote (blindly) a first draft for the report descriptor: // 2 bytes for buttons + padding, 4 bytes for X, Y, Rx, Ry
#define SC_HID_GAMEPAD_EVENT_SIZE 6
const uint8_t SC_HID_GAMEPAD_REPORT_DESC[] = {
// Usage Page (Generic Desktop)
0x05, 0x01,
// Usage (Gamepad)
0x09, 0x05,
// Collection (Application)
0xA1, 0x01,
// Collection (Physical)
0xA1, 0x00,
// Usage Page (Buttons)
0x05, 0x09,
// Usage Minimum (1)
0x19, 0x01,
// Usage Maximum (15)
0x29, 0x0E,
// Logical Minimum (0)
0x15, 0x00,
// Logical Maximum (1)
0x25, 0x01,
// Report Count (15)
0x95, 0x05,
// Report Size (1)
0x75, 0x01,
// Input (Data, Variable, Absolute): 15 buttons bits
0x81, 0x02,
// Report Count (1)
0x95, 0x01,
// Report Size (1)
0x75, 0x01,
// Input (Constant): 1 bit padding
0x81, 0x01,
// Usage Page (Generic Desktop)
0x05, 0x01,
// Usage (X)
0x09, 0x30,
// Usage (Y)
0x09, 0x31,
// Usage (Rx)
0x09, 0x33,
// Usage (Ry)
0x09, 0x34,
// Logical Minimum (0)
0x15, 0x81,
// Logical Maximum (255)
0x25, 0xFF,
// Report Size (8)
0x75, 0x08,
// Report Count (4)
0x95, 0x03,
// Input (Data, Variable, Absolute): 4 bytes (X, Y, Rx, Ry)
0x81, 0x02,
// End Collection
0xC0,
// End Collection
0xC0,
};It assumes that the thumbsticks use X, Y, Rx, Ry (but I'm not sure this is correct). I plan to implement what's needed to handle HID game controllers (AOA and UHID), so that I can test and debug. I don't know when though, but that's probably my next subject on scrcpy. |
|
This simulates an Xbox One controller: Descriptor: Input report format:
To enable rumble (vibration), the vendor ID and product ID must be 0x045e and 0x02fd for a special driver to load, it sends output reports for rumble:
|
|
Thank you for the documentation! Will be useful 👍
So the XBoxOne controller uses X, Y for the left stick and Z, Rz for the right stick. This is also what I have seen here (also the report descriptor is a bit different). Report descriptors for other controllers also seem to use the same 👍 |
|
Implemented in #5270. Feedback welcome 😉 |
closes #99
What it does
Important stuff
open,ioctl,writeandclose). Since we can't use most of these directly from Java (AFAIK), we need either native code or a library that allows us to call these functions. To avoid depending on the NDK, I'm using the Java Native Access library;app_processdoesn't want to load it, so right now it's extracted to/data/local/tmpon initialization and cleaned up later.Things you can help with
ioctlsyscall without native code;To-do
Out of scope
I probably won't be adding some special features to this pull request, such as:
Feedback
If you have any feedback (suggestions, complaints, new info, ...), please don't hesitate to write it down below.
Tested devices
The list of (un)supported devices can change anytime.
Reporting a new device
If you have tested with a device not on the list, please report it by writing a comment below. Please follow (more or less) the following format:
NEW: Windows and Linux releases
Prebuilt versions of scrcpy with the new patches can be found here. Remember to set the
SCRCPY_SERVER_PATHvariable.